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1.  Summary 


Air  Force  AF05-071,  “Embedding  Cognitive  Systems  into  Systems  Engineering  Practice,”  had 
the  following  objective:  “Develop  a  process  and  toolset  to  embed  the  application  of  the 
emerging  practices  and  technologies  of  cognitive  systems  into  the  traditional  practice  of  systems 
engineering.” 

The  topic  was  interpreted  to  address  a  pernicious  and  tenacious  problem.  The  failure  to  include 
human  characteristics,  needs,  and  cost  influences  within  the  development,  operations  and 
sustainment  cycle  was  resulting  in  failed  systems,  products  people  refused  to  use,  and  soaring 
operations  costs.  Research  showed  the  problem  had  been  around  for  at  least  50  years;  we  believe 
the  duration  is  closer  to  150  years.  It  is  intensifying  as  the  prominence  and  ubiquity  of  software- 
driven  products  and  systems  increases.  We  resolved  to  use  the  contract  opportunity  to  retire  the 
problem.  Challenges  to  retiring  the  problem  were  two-fold.  They  can  be  related  to  the  process 
and  the  toolset. 

1.1.  Challenges 

Process  Challenges.  Three  separate  contractors  were  ostensibly  developing  three  different 
processes.  If  the  develop  processes  were  not  identical,  which  would  be  deemed  correct?  Would 
someone  be  forced  to  pick  and  choose  from  the  offerings  to  select  one  in  whole  or  several  parts 
that  would  somehow  be  later  integrated?  Additionally,  neither  the  contractors  nor  the  sponsor 
owned  the  systems  engineering  process.  Several  organizations  or  professional  societies,  the 
International  Standards  Organization  (ISO),  IEEE,  and  the  International  Council  on  Systems 
Engineering  (INCOSE),  have  produced  standards  and  handbooks  which  authoritatively  capture 
the  systems  engineering  process.  It  is  upon  these  standards  that  the  Defense  Acquisition  System 
(DAS)  squarely  rests.  If  the  problem  were  to  be  retired,  changes  to  authoritative  documentation 
would  need  to  be  agreed  upon  and  adopted. 

Toolset  Challenges.  Post-phase-II  commercialization  of  the  product  or  service,  in  this  case  the 
toolset,  is  a  goal  of  the  SBIR  program.  Investigations  revealed  an  absence  of  a  market  for  a 
toolset.  In  part  this  was  because  of  interpretations  of  the  term  “toolset”  that  differed  between 
disciplines.  In  part  this  was  due  to  the  lack  of  a  recognized  need. 

To  a  systems  engineer,  a  toolset  is  a  collection  of  software  that  aids  in  analysis,  definition  and 
management;  simulations  and  requirements  and  configuration  management  software  are 
examples.  To  a  cognitive  engineer  or  a  human  systems  integrator,  a  toolset  is  a  collection  of 
methods  or  procedures  for  performing  knowledge  elicitation,  task  analyses,  or  workload  and 
manpower  assessments,  as  examples.  Contrast  a  systems  engineering  simulation,  a  computer 
program  that  performs  mission  analysis  in  support  of  requirements  development  with  a  cognitive 
walkthrough,  a  verbal  simulation  in  which  a  storyboarded  scenario  undergoes  stepwise  review 
and  the  situation  is  clearly  revealed.  The  first  is  an  electronic  product,  and  the  second  is  an 
interpersonal  communication. 

Cognitive  engineering  methods  do  not  lend  themselves  to  coding.  They  are  interactive 
investigations  between  engineer  and  subject.  Practicing  cognitive  engineers  have  their  own  tools 
which  they  evolve  and  refine.  They  are  not  in  the  market  for  a  software  product. 
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Systems  engineers,  on  the  other  hand,  are  interested  in  software  products.  Most  practitioners  are 
unaware  of  the  definition,  scope  of  application,  and  limitations  of  cognitive  engineering.  They 
are,  for  the  most  part,  not  aware  of  the  need  for  cognitive  engineering.  Contractual  cognitive 
engineering  requirements  are  rarities;  when  they  do  appear,  they  are  not  considered  system 
drivers. 

Thus  the  greatest  challenge  for  the  toolset  was  not  to  develop  one,  but  to  create  a  market  for  one. 
Without  awareness  and  demand,  any  toolset  created  would  languish. 

1.2.  Two-Fold  Solution 

A  two-pronged  solution  was  developed  to  address  the  two  challenges.  The  first  was  a 
noncontract  solution,  a  company  investment  in  the  founding  and  development  of  the  INCOSE 
Human  Systems  Integration  Working  Group  (HSIWG).  The  contracted  work  represented  the 
second  solution.  This  was  the  development  of  the  Acquisition  Practitioner  Support  Environment 
(APSE),  a  free  toolset  for  bringing  together  the  acquisition  stakeholders  necessary  to  achieving 
the  implicit  topic  goal. 

1.2.1.  HSIWG 

The  lack  of  awareness  and  demand  for  cognitive  engineering  extends  to  program  and  project 
managers.  These  acquisition  stakeholders  have  little  incentive  to  include  cognitive  engineering 
scope  in  program  plans.  We  found  that  this  situation  is  not  unique  to  cognitive  engineering. 
Project  managers,  as  represented  by  Project  Management  International  (PMI),  are  working  to 
document  their  value  to  their  customers  (1).  Systems  engineering  practitioners  complain  they  are 
hired  too  late  and  are  perceived  to  add  major  cost  with  little  value.  The  importance  of  the 
contributions  of  human  systems  engineers  are  increasingly  being  recognized  by  the  owners  of 
capabilities  (systems),  but  owners  are  unable  to  articulate  this  value  in  an  executable  fashion. 


Figure  1 :  Acquisition  stakeholders  —  self-isolated  from  users  and  from  one  another 
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Figure  1  represents  the  situation.  Each  of  the  stakeholders  addressed  in  this  project  -  program 
and  project  managers,  systems  engineers,  human  system  integrators  and  cognitive  engineers 
move  in  their  own  circles.  Their  work,  their  research  and  publications  are  inwardly  focused  on 
their  own  needs,  their  esoteric  professional  jargon,  and  their  own  communities  of  practice.  This 
is  particularly  damaging  for  the  human  systems  integration  (HSI)  disciplines  because  many  of  its 
contributing  domains  -  manpower,  personnel,  training,  human  factors,  occupational  health  and 
safety  -  are  separate  disciplines  unto  themselves.  We  found  that  in  the  case  of  training,  for 
example,  practitioners  of  that  discipline  are  unaware  of  their  relationship  to  the  integrating 
discipline,  and  to  the  developmental  tradeoffs  that  impact  their  operational  influence  and 
contributions  to  total  ownership  cost. 

The  view  in  figure  1  is  the  20th  Century  situation  in  miniature.  The  20th  century  was  the  age  of 
specializing.  Specialists  worked  independently  to  improve  the  practice  of  their  discipline,  its 
influence,  and  its  employment  rates.  These  are  the  goals  of  the  20th  century  professional 
organization. 

The  HSIWG  was  fonned  to  provide  a  forum  wherein  interdisciplinary  discussions  could  take 
place.  Systems  engineers,  members  of  the  host  organization  engaged  human  factors  engineers, 
human  systems  integration  practitioners,  cognitive  engineers  and  members  of  IEEE  Systems, 
Man  and  Cybernetics.  Interdisciplinary  and  cross-organizational  exchanges  occurred.  Member 
representatives  from  the  Air  Force  Human  Systems  Integration  Office  and  its  71 1th  Command 
Wing,  MANPRINT  (U.S.  Army)  and  the  Navy’s  human  systems  integration  office  opened 
bidirectional  lines  of  communication. 

The  organization  began  by  developing  a  charter,  a  vision  statement  and  a  definition  of  human 
systems  integration.  Body  material  and  a  human  systems  integration  appendix  was  submitted  to 
and  accepted  for  the  INCOSE  Systems  Engineering  Handbook  version  3.0  (2).  The  IEEE  1220 
systems  engineering  standard  (3),  a  more  abstract  document,  already  included  human  systems 
engineering  throughout  its  text.  This  meant  that  authoritative  documents  from  the  largest 
professional  organizations  representing  systems  engineers  prescribed  the  inclusion  of  human 
systems  integration  in  the  systems  engineering  process.  This  was  a  step  forward,  but  did  not 
serve  to  institutionalize  the  incorporation  of  human  systems  integration  into  systems  engineering 
or  into  the  systems  engineering  process. 

Defense  Acquisition  Systems  documentation,  as  represented  on  the  Defense  Acquisition 
University  web  site  (4),  called  for  human  systems  integration  at  the  top  level,  but  there  remained 
no  documentation  on  or  community  of  practices  for  human  systems  integration  or  cognitive 
engineering.  Additionally,  senior  systems  engineers,  those  in  the  position  to  shape  the 
implemented  process  do  not  feel  the  need  to  read  handbooks  or  standards.  They  rely  on  their 
experience  and  customer  requirements  to  guide  their  practice. 

To  address  this,  this  contract’s  principal  investigator  co-edited  a  special  edition  of  INCOSE 
Insight  magazine  (5),  a  non-juried  magazine  designed  to  quickly  move  best  practices  into  the 
field.  An  article  from  the  JPRINT  office  was  included  to  provide  descriptions  of  the  services’ 
human  systems  integration  initiative  as  well  authoritative  statements  of  human  systems 


3 


integration  requirements  that  systems  engineers  would  be  encountering  with  ever  greater 
prevalence  in  future  procurements. 

We  assert  Human  Systems  Integration  is  the  conduit  by  which  cognitive  engineering  will 
penetrate  acquisition  programs.  Cognitive  engineering  contributes  to  the  manpower,  personnel, 
training  requirements  generation,  habitability,  survivability,  and  human  factors  components  of 
human  systems  integration.  Therefore  a  second  special  edition  of  Insight  magazine,  this  one 
with  a  cognition  theme,  is  in  process  with  an  April,  2009  publication  date. 

1.2.2.  APSE 

The  HSIWG  was  fonned  to  create  a  climate  for  the  process  required  by  the  topic;  it  was  to  open 
a  market  for  a  toolset.  APSE  is  a  web-enabled  software  application  that  points  up  opportunities 
for  joint  practice  among  the  system  owner’s  program  manager,  the  contractor’s  project  manager, 
and  systems  engineers,  human  systems  integrators  and  cognitive  engineers  representing  the 
owner  as  well  as  those  on  the  contractor  team. 
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The  integrated  Master  Plan 
(IMP)  it  a  contractor 
prepared  customer-required 
tool  that  rs  used  to  track  and 
measure  projeclTask 
accomplishment  It  identrftes 
essential  events  program 
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technical  tasks  and  technical 
development  and  other  nsk 
reduction  actwties  The  IMP 
consists  of  a  table  indexed  to 
the  Work  Breakdown 
Structure  and  a  descrtptr* 
narrative  of  execution 
processes  objectwes  and 
control  documents  The  IMP 
is  an  event -based  ptan  and 
contractually-binding 
document  that  is  relatively 
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Mature  technologies  and  fit  «ito  concept 


Figure  2:  APSE  main  page 

The  APSE  main  page  is  shown  in  figure  2.  The  APSE  interface  was  designed  to  be  interesting 
and,  to  an  extent,  entertaining.  This  was  to  attract  users  and  to  encourage  them  to  linger,  to 
explore  the  material  contained  in  the  tool. 
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APSE  text  fields  were  purposefully  word  constrained.  We  found  people  were  overloaded  by 
weighty  process  documents  and  text  books  of  systems  engineering  or  task  analysis.  While  one 
may  assert  that  reading  these  documents  is  a  necessary  part  of  their  jobs,  we  observed  that  people 
were  not  familiar  with  the  material. 

APSE  assumes  the  six -phase  DAS  as  the  baseline  system  life  cycle  process.  The  Joint 
Capabilities  Identification  and  Development  (JCIDS)  process  was  added  as  a  seventh,  precursor 
phase  because  of  the  importance  of  including  human  engineering  during  gap,  needs,  and  solution 
analyses.  When  the  phase  is  selected,  a  short  description  of  the  phase’s  purposes  is  displayed 
along  the  bottom  bar. 

For  each  of  the  seven  phases  shown  on  the  left,  five  in-process  deliverables  were  selected.  In  the 
figure  2  example,  the  System  Development  Phase  includes  the  five  deliverables  WBS/IPT/CPT, 
Prime  Contract,  User  Interface,  Trade  Studies,  Integrated  Master  Plan  /  Schedule.  Our 
investigations  showed  that  systems  will  deliver  better  performance  and  lower  total  ownership 
costs  because  human  systems  integrators  and  cognitive  engineers  have  contributed  to  these 
thirty-five  deliverables. 

The  main  page  contains  a  succinct  description  of  the  product  (first  white  field  from  the  left)  and 
a  brief  description  of  the  integration  opportunity  the  deliverable  affords  for  interdisciplinary 
interaction  (white  field  on  right).  These  boxes  allow  scrolling  when  even  a  short  explanation 
exceeded  the  field’s  size.  Like  many  of  the  interface’s  fields,  the  box  expands  when  clicked  so 
the  whole  field  can  be  seen  at  a  glance.  The  font  size  grows  with  the  expansion  as  well;  this  was 
a  useful  feature  when  investigators  briefed  the  tool. 

The  accordion  fields  immediately  to  the  left  of  the  Integration  Opportunity  contain  the  most 
important  tasks,  estimates  of  man-time,  and  applicable  methods  and  software  for  each  of  the  five 
APSE  customers.  Tasks  were  selected  to  acquaint  users  with  what  they  should  be  doing,  but 
more  importantly  what  other  contributors  are  doing.  Man-time  is  presented  graphically  as 
relative  estimates;  these  were  included  to  address  project  managers’  concerns  that  the  addition  of 
human  systems  integrators  and  cognitive  systems  engineers  would  break  their  budgets.  Methods 
help  practitioners  to  develop  a  common  lexicon;  systems  engineers  did  not  know  what  to  call  the 
activities  of  cognitive  engineers.  Providing  a  reference  enables  them  to  consult  on-line  or  textual 
references  to  leam  more  when  they  need  to. 

Software  recommendations  are  meant  to  be  exemplary  of  what  is  available  to  support  the 
activities  of  APSE  customers.  At  the  beginning  of  phase  II,  we  hypothesized  those  tools 
necessary  to  support  the  embedment  of  cognitive  systems  into  systems  engineering  practice 
already  existed.  We  set  out  to  identify  those  tools  and  demonstrate  some  so  APSE  users  could 
see  how  these  software  aids  could  be  used  as  interdisciplinary  communication  devices. 

APSE  users  seeking  more  detail  can  select  a  deliverable  from  the  menu  bar  at  top.  They  are  then 
taken  to  the  detail  deliverable  page  (figure  3).  On  the  top  at  the  left,  the  deliverable  page 
provides  man-time  estimates  for  instances  when  the  deliverable  is  being  developed  as  part  of  a 
major  acquisition  and  for  cases  when  the  activity  is  being  done  as  part  of  a  focused  effort.  Input 


5 


and  output  conditions  are  provided  for  purposes  of  technical  planning  and  earned  value  cost 
management. 

On  the  right,  Setting  describes  where  the  project  or  product  is  in  its  life  cycle,  what  has  been 
done  and  what  needs  to  be  done  to  generate  the  deliverable.  Assumptions,  below,  describe  what 
APSE  investigators  took  for  granted  when  putting  together  the  deliverables  page  materials  - 
often  this  field  provided  background  for  man-time  estimates. 

At  the  bottom,  more  expanded  listings  of  tasks,  methods  and  software  are  provided  for  each 
contributing  discipline.  This  enables  practitioners  to  get  a  more  complete  picture  of  their 
responsibilities  at  this  point  in  the  life  cycle. 
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way  people  are  using  it 

Tasks  are  properties  of  the  mission,  system  and  its  hardware, 
software  and  hunan  components  as  opposed  to  being  items  on  the 
protects  integrated  master  schedule  In  the  military,  a  functional 
Area  Analysis  is  performed  to  identify  operational  tasks,  a  Frictional 
Needs  Analysis  is  performed  to  assess  the  ability  to  accomplish 
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Figure  3:  APSE  deliverable  page 


Software  was  demonstrated  for  some  of  the  most  important  human  systems  integration  and 
cognitive  engineering  activities.  Table  1  provides  the  software  demonstrated  and  why  it  was 
selected. 

Table  1 :  Software  selected  for  demonstration 


Software 

Why  software  was  selected 

Micro 

Saint 

Sharp 

Systems  engineers  would  appreciate  a  simulation  aid.  The  product  can  be  used 
to  depict  scripted  human  interactions  with  products.  Its  engine  underpins  the 
Imprint  human  systems  integration  tool. 

TestLog 

Demonstrated  human  aspects  of  product  testing  that  need  to  be  documented  as 
part  of  test  planning,  overseen  during  text  execution  and  documented  as  part  of 
post-test  analysis. 

Task 

Architect 

Task  bookkeeping  software  useful  for  capturing  results  of  behavior  and 
cognitive  task  analyses.  Analogous  to  the  systems  engineering  tools  Doors, 

Core  or  Slate. 
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We  used  a  product  developed  as  part  of  another  SBIR  effort  to  document  our  experience  with  the 
software  demonstration.  The  Geeksee  interface  is  built  upon  the  foundations  of  the  abstraction- 
decomposition  matrix  of  the  cognitive  work  analysis  approach  to  cognitive  engineering. 

Purposes,  measures  and  goals  are  at  the  top,  functions  are  in  the  middle  moving  downward  and 
objects  and  details  are  at  the  bottom.  Figure  4  provides  an  example  for  the  TestLog  software;  the 
same  format  was  used  to  document  the  results  of  all  software  demonstration 

On  the  left,  the  purposes,  functions  and  a  link  to  the  manufacturer’s  web  page  are  listed.  Purpose 
statements  were  written  by  project  investigators.  Functions  were  generally  taken  from 
manufacturer’s  literature.  On  the  right,  the  goals  of  the  software  demonstration  are  given  at  the 
top.  The  demonstration  steps  are  listed  in  cookbook  fashion  in  the  middle  panel  moving  down. 
Each  step  has  an  associated  artifact  which  is  displayed  in  the  center  window  when  the  step  is 
clicked  by  the  user.  In  the  example,  the  TestLog  page  for  the  unit-level  test  has  been  clicked  and 
the  test  description,  anticipated  results,  actual  results  and  analysis  are  shown.  The  center 
window  expands  to  fill  the  display  screen  when  selected  allowing  viewers  to  scroll  through  the 
material. 
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Figure  4:  TestLog  software  as  example  of  software  aid  page 
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For  planning  purposes,  we  have  included  costs  of  using  the  software  at  the  top.  Investigators 
noted  the  time  it  took  to  execute  each  of  the  software  setup  and  demonstration  steps.  This  time  is 
included  in  graphic  fonn.  Below  that,  the  price  of  the  software  is  given.  Prices  are  indicated  by 
range  so  people  thinking  of  purchasing  the  software  know  if  it  is  around  $100  or  in  the  $5,000  to 
$10,000  range  for  example. 

At  the  bottom,  is  a  link  to  a  document  that  describes  our  experience  in  using  the  tool.  The 
experience  reports  describe  glitches,  workarounds,  and  lessons  learned.  Some  entries  suggest 
improvements  that  would  make  the  software  more  effective  or  usable. 

1.2.3.  Summary  Findings  and  Conclusions 

Figure  5  shows  the  target  end  state  for  the  effort.  In  this  vision,  government  and  contractor 
program  managers,  project  managers,  systems  engineering,  human  systems  integrators  and 
cognitive  engineers  work  together  alongside  the  specialty  engineers  -  mechanical,  electrical, 
civil,  industrial,  quality,  etc.  At  the  center  of  their  considerations  are  the  jobs  users  must  do  - 
manufacture,  test,  train,  operate,  maintain,  supply,  and  train. 

The  work  done  to  bridge  disciplinary  boundaries  was  successful  within  limits.  INCOSE  is 
working  toward  establishing  a  memorandum  of  understanding  with  Human  Factors  and 
Engineering  Society.  HSIWG  representatives  have  been  encouraged  and  empowered  to  engage 
the  American  Society  of  Safety  Engineers.  Meetings,  sponsored  by  the  cognitive  engineering 
community,  got  acquisition  personnel  and  those  experienced  with  product  development  to  sit 
across  from  researchers,  trying  to  reach  a  meeting  of  the  minds.  Systems  engineers  have 
participated  in  cognitive  engineering  brown  bag  discussions  on  a  monthly  basis.  We  have  shared 
the  figure  5  vision  with  training  and  information  technology  providers.  Papers  have  been  written 
and  presentations  made. 


Figure  5:  Acquisition  stakeholders  collaborating,  users  receive  focus 

Nevertheless,  practice  remains  personality  based.  The  job  of  re-engineering  a  culture  involves 
changing  one  mind  at  a  time.  People  have  to  be  at  the  table  when  discussions  are  held.  It  is  not 
enough  to  write  papers,  practitioners  need  to  recognize  the  applicability  to  their  work  and  the 
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value  they  provide.  When  investigators  presented  APSE  at  the  EITSEC,  passersby  were  asked, 
“Does  your  work  involve  humans?”  Most  of  the  people  responded,  “No,  I  don’t  do  anything  that 
has  to  do  with  people.”  ‘I  write  middleware.’  ‘I  do  blast  analysis.’  There  is  no  recognition  that 
someone  is  going  to  use  that  middleware  to  get  two  systems  talking  to  one  another,  and  that  the 
middleware  can  be  written  so  that  the  linking  is  easy,  efficient  and  effective  or  impossibly 
complex  and  time  consuming.  Eventually  that  middleware  will  need  to  be  updated,  that  task, 
too,  could  be  easy  to  do  or  difficult.  The  design  of  the  product  and  the  user  interface  will  have 
an  impact  on  productivity  and  competitiveness.  In  military  terms,  this  translates  into  superiority. 
The  same  can  be  said  for  the  blast  analysis.  Someone  will  have  to  apply  those  data.  How  will  it 
be  used?  What  decisions  will  be  based  on  the  results? 
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Figure  6:  The  missing  link  when  design  is  infonned 


We  established  there  was  a  function  missing  from  the  cognitive  engineering  to  software 
manufacture  process.  In  our  effort,  we  supplied  that  piece  with  a  graphic  designer.  More 
properly,  for  mission  critical  interfaces,  that  person  is  a  human  computer  interface  designer.  The 
HCI  expert  appears  to  have  the  skills  to  translate  user  requirements  developed  by  cognitive 
engineers  into  design  specifications.  This  is  the  function  responsible  design  engineers  provide 
for  systems  engineers. 

This  finding  was  illuminating  to  the  members  of  the  cognitive  engineering  community  with 
whom  we  interacted.  It  solved  some  of  the  riddles  that  were  discovered  in  phase  I  about  why 
software  engineers  were  unable  to  use  the  products  of  cognitive  engineering. 

This  was  determined  to  be  only  part  of  the  answer.  Cognitive  engineers  also  needed  to  learn 
about  technical  management  processes,  such  as  configuration  management,  in  order  to  insert  the 
requirements  they  discover  into  the  development  process.  Configuration  management  is 
particularly  important  to  a  discipline  that  is  continually  engaged  late  in  the  cycle.  For  this 
reason,  APSE  includes  technical  management  prominently  among  the  deliverables  it  includes. 

APSE  is  finished,  but  not  complete.  We  believe  its  value  would  benefit  from  additional 
marketing  activities,  extension  to  other  deliverables,  additional  software  demonstration,  and  from 
placement  with  prominent  acquisition  forums  such  as  the  Defense  Acquisition  University. 
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2.  Introduction 


This  is  the  final  report  for  Air  Force  Small  Business  Innovative  Research  (SBIR)  topic  AF05- 
071.  The  objective  of  “Embedding  Cognitive  Systems  into  Systems  Engineering  Practice,”  was 
to  “develop  a  process  and  toolset  to  embed  the  application  of  the  emerging  practices  and 
technologies  of  cognitive  systems  into  the  traditional  practice  of  systems  engineering.”  Contract 
FA8650-06-C-6638,  an  effort  that  extended  from  April,  2006  through  December,  2008,  was  one 
of  three  phase  II  contracts  let  for  Air  Force  topic  AF05-07 1 .  The  other  two  contracts  were  let  to 
Aptima  and  CHI  Systems,  Inc.  This  report  refers  to  the  work  of  these  two  contractors,  but  does 
not  report  on  their  activities. 

During  2006  and  2007,  four  hundred  fifty  man-hours  were  invested  in  founding  and  starting  the 
INCOSE  HSIWG.  The  HSIWG  is  a  mechanism  for  creating  a  market  for  the  process  and 
products  developed  under  this,  and  the  other  two,  contracts.  Human  systems  integration  paves 
the  way  for  acceptance  of  cognitive  engineering  into  systems  engineering  processes.  The 
HSIWG  activity  was  not  part  of  our  proposed  work  plan;  however,  contract  dollars  were 
expended  to  support  events  that  were  educational  for  HSIWG  meeting  attendees.  We 
incorporated  information  presented  at  those  seminars  into  APSE,  the  product  of  the  extant 
contract  effort.  Thus,  the  contract  and  noncontract  activities,  while  severable,  are  intertwined. 
HSIWG  activities  are  pertinent  to  this  report  and  are  included  in  its  next  four  sections. 

Section  3  describes  the  methods,  assumptions  and  procedures  used  in  our  attempt  to  infuse 
systems  engineering  with  human  systems  engineering  and,  most  particularly,  cognitive  systems 
engineering.  It  describes  how  we  arrived  at  the  product  contents  and  the  steps  taken  to  open  the 
market  for  the  required  process  and  toolset. 

Section  4  reports  on  the  results  of  our  efforts.  It  describes  the  effects  of  our  social  engineering 
attempts  to  bring  multiple  disciplines  and  communities  together  and  the  end  state  as  of  the 
contract’s  conclusion.  It  details  the  features  of  the  APSE  product  and  our  success  in  bringing  it 
to  market. 

Section  5  provides  conclusions,  a  reflection  on  our  results,  the  contract  scope  and  the  nature  of 
the  enduring  problem  of  marrying  hard  and  soft  engineering. 

Section  6,  Recommendations,  describes  what  remains  to  be  done  and  suggests  a  road  ahead. 
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3.  Methods,  Assumptions  and  Procedures 

3.1.  Overview 

“Quite  often,  I  somehow  hit  a  combination  of  keys  that  summons  a  box  that  says,  in  effect,  ‘This  is  a 
Pointless  Box.  Do  You  Want  It?’  which  is  followed  by  another  that  say  ‘Are  You  Sure  You  Don’t 
Want  the  Pointless  Box?’  Never  mind  all  that.  I  have  known  for  a  long  time  that  the  computer  is  not 
my  friend.”  -  Bill  Bryson  in  I’M  A  STRANGER  HERE  MYSELF  (6). 

“The  federal  government  has  spent  $195  million  on  a  long -promised  wireless  radio  network  for  the 
nation's  law  enforcement  agencies  that  is  at  "high  risk  of  failure,"  the  Justice  Department's  inspector 
general  reported  yesterday.  Inspector  General  Glenn  A.  Fine  blamed  delays,  funding  shortfalls 
and  infighting  among  the  Justice,  Homeland  Security  and  Treasury  departments,  whose  81,000  agents 
are  expected  to  use  the  $5  billion  system  when  it  is  completed  by  2021.”  -  Spencer  S.  Hsu  and  Charles 
Babington,  Washington  Post,  Tuesday,  March  27,  2007  (7). 

“The  set  of  people  who  are  frustrated  every  day  by  badly  designed  information  technology  is  very 
large.”  -  K.J.  Vicente,  “Crazy  Clocks:  Counterintuitive  Consequences  of  ‘Intelligent’  Automation.”  (8) 

“As  with  many  large-scale  projects  there  were  many  stakeholders  to  manage  both  at  the  project  and 
operational  levels.  This  project  was  no  exception  and  the  contractual  arrangement  between  stakeholder 
organizations  further  added  to  the  complication  of  human  factors  integration.”  -  lan  Rowe,  “Practical 
Human  Factors  Integration,  Lessons  Learnt  from  a  case  study  of  large  project  implementation.  (9) 

“When  NCR  Corp.  asked  executives  how  they  deal  with  the  increasing  amount  of  data  generated  by 
the  corporate  world  each  day,  it  sounded  as  though  the  executives  need  a  life  raft.  Executives  of 
companies  large  and  small  spoke  of  swimming  or  drowning  in  data.  Others  said  they  feel  frozen, 
unable  to  make  confident  decisions  when  numbers  conflict  or  take  too  long  to  arrive  in  an 
understandable  form.”  -  Shannon  Joyce  Neal,  “Today’s  executive  swims  in  a  sea  of  numbers.”  (10) 

“For  operations  in  civil  airspace,  the  term  autonomous  civil  aircraft  implies  the  ability  to  perform  all 
the  typical  functions  required  for  safe  flight  while  flying  in  conformance  with  national  airspace 
constraints,  without  having  a  human  in  the  control  loop,  either  on-  or  off-board.”  -  Herman  A.  Rediess, 
and  Sanjay  Garg,  “Autonomous  civil  aircraft  —  the  future  of  aviation?”  (11) 


Figure  7:  First  there  was  the  automobile;  then  there  were  passengers  (12) 

The  above  examples  provide  a  small  sampling  of  the  dissatisfaction,  failure,  and  frustration  that 
result  from  poor  integration  of  human  needs,  aptitudes  and  costs  into  developing  products  and 
their  operation.  We  seem  to  treat  people  as  in  the  old  Hertz  Rental  Car  advertisements  -  the 
system  is  there  and  the  people  are  dropped  in  (figure  7).  Resultant  costs  are  high  measured  in 
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dollars  invested  in  failed  systems,  productivity  losses  in  the  workplace,  and  lives  lost  on  the 
battlefield.  This  problem  is  not  new. 

Phase  I  research  revealed  that  the  problem  of  including  human  characteristics,  needs,  and  cost 
influences  within  the  development,  operations  and  sustainment  cycle  has  been  around  for 
between  30  and  50  years.  It  has  probably  been  around  much  longer.  We  assert  that  the  problem 
arose  with  the  beginning  of  the  industrial  revolution  when  conception  and  construction  of  tools 
became  the  responsibility  of  people  other  than  the  users  of  the  tools. 

When  smithies  and  carpenters  crafted  their  own  instruments,  tools  were  designed  by  their  users. 
Tools  were  constructed  by  the  artisans  themselves  to  address  the  needs  of  their  trade.  These 
artisans  had  the  skills  to  modify,  redesign  and  reconstruct  their  tools  and  so  articles,  like  the 
hammer,  evolved  to  the  needs  of  the  tasks  they  supported.  When  artisans,  such  as  candle 
makers,  did  not  have  the  skills  necessary  to  build  their  own  tools,  they  had  access  to  the 
specialists  to  whom  they  could  relate  detailed  instructions,  in  person,  for  the  construction  of  the 
gear  they  needed. 

As  the  scope  of  tools  increased  from  hand-held  tools  to  compound,  powered  machines,  the 
fascination  with  technological  novelty  began  to  drive  the  imaginations  of  specialists  and  to  wag 
the  tail  of  product  development.  When  market  forces  prevail,  customer  demand  reigns  in  flights 
of  imagination  -  products  that  better  satisfy  needs  and  tastes  survive  because  they  are  purchased. 
Market  research  is  performed  to  align  development  investment  with  customer  preference. 

When  the  developer-user  relationship  is  moderated  by  a  purchasing  organization  or  virtual 
monopoly  which  restricts  customer  choice,  as  in  the  Bryson  example  above,  the  user  becomes 
little  more  than  a  ghost.  The  forces  that  would  drive  out  innovation  for  the  sake  of  novelty  are 
absent.  People  like  CIOs  decide  what  users  need  and  their  opinions,  not  user  demand,  dictate 
system  characteristics.  What  should  be  an  assessment  of  functional  need,  becomes  a  battle  of 
wills.  In  the  case  of  IWN,  $195  million  was  placed  at  risk. 

The  need  to  achieve  the  integration  of  human  systems  integration  (HSI)  with  the  acquisition 
process  has  intensified  with  the  prominence  and  ubiquity  of  software-driven  products  and 
systems.  NCR  executives  reported  this  in  the  article  cited  above;  it  is  universal.  Autonomous 
systems  can  make  matters  worse.  Unintuitive  actions  taken  by  automated  systems  can  surprise, 
confuse  and  frighten  users;  when  users  are  required  to  invoke  manual  override  or  implement 
corrective  action,  the  lack  of  insight  can  lead  to  disaster.  This  was  an  issue  for  the  Navy’s  DDX 
program  which  sought  manpower  reductions  through  automation. 

Investigators  for  this  project  set  out  to  retire  this  enduring  and  escalating  problem.  We  proposed 
an  aggressive  work  plan.  The  work  plan  was  modified  dynamically  to  take  advantage  of  what 
we  learned  from  our  research  and  identified  opportunities.  Our  policy  was  to  share  what  we 
learned,  to  connect  individuals  and  communities,  to  educate  ourselves  in  public  forums,  and  to 
spread  the  learning  as  broadly  as  possible,  to  give,  give,  and  give  some  more. 
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3.2.  The  Work  Plan  -  Beginning  and  End  States 

The  proposed  work  plan  was  integrative.  The  task  names  and  purpose  are  given  in  table  2.  Note 
that  software  tools  are  referred  to  as  aids.  This  is  because  the  tenn  “tools”  has  a  different 
meaning  for  cognitive  engineers  and  scientists,  to  whom  a  tool  is  a  method  or  procedure,  than  it 
does  for  systems  engineers  to  whom  tools  are  software  products. 


Table  2:  Proposed  Work  Plan 


# 

Title 

Purpose 

1 

Identify  Aids  that  Support  An 
Effective  Set  of  Cognitive 
Engineering  Activities. 

Cognitive  engineering  activities  were  identified. 

Software  aids  that  helped  in  their  completion  were 
investigated.  A  gap  analysis  was  perfonned. 

2 

Investigate  Commercial  Data 
Management  Products 

A  loosely  integrated  confederation  of  software  tools  was 
envisioned.  Commercial  products  were  investigated  to 
bring  the  software  aids  together. 

3 

Accommodate  Acquisition 
Software  Aids 

The  toolset  to  embed  cognitive  engineering  was  to 
exchange  data  with  software  aids  for  systems  engineering 
and  cost,  schedule  and  risk  estimation. 

4 

Demonstrate  Value  to  Decision 
Makers 

Specialty  tools  were  envisioned  to  guide  the  program  and 
project  managers  and  systems  engineers  to  understand 
when  to  interact  with  cognitive  engineers  and  when 
cognitive  engineers  were  most  needed. 

5 

Software  Development 

Architecture,  features,  functions  and  support  to  five 
stakeholders  was  defined.  Plans  were  made  to  develop, 
demonstrate  and  test  the  new  software. 

6 

Verification  and  Validation 

This  task  was  to  confirm  our  results. 

7 

Reporting 

We  apprised  our  sponsor  stakeholders  of  our  progress 
and  findings. 

8 

Awareness,  Appreciation  and 
Acceptance 

This  is  essential  a  marketing  task  designed  to  move  the 
product  into  use  and  satisfy  SBIR  phase  III  requirements. 

Because  we  adopted  processes  that  incorporated  continuous  learning,  an  approach  favored  by 
cognitive  engineers,  the  proposed  work  plan  evolved  during  the  period  of  perfonnance.  Figure  8 
shows  the  way  in  which  the  work  plan  was  aligned  at  the  project’s  conclusion.  Most  of  the  table 
2  tasks  are  recognizable  in  figure  8,  but  there  is  not  a  one-to-one  correspondence.  The  purposes 
of  some  tasks  changed,  though  the  titles  remained  appropriate.  Other  tasks  were  merged  or 
transformed.  Some  tasks  were  added.  The  figure  shows  how  the  noncontract  HSIWG  activities 
fit  in  to  work  plans.  The  reporting  task  is  not  shown  in  figure  8,  but  was  executed. 

3.2.1.  Overview  of  Figure  8 

The  effort  divided  into  three  prongs.  On  the  left,  we  researched  and  developed  the  process  and 
tool  set.  On  the  right,  market  development  activities  are  shown.  In  the  center  is  a  bridging 
activity  in  which  we  specifically  marketed  our  product. 
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Figure  8:  Work  plan  at  project’s  conclusion 


APSE,  the  toolset,  was  envisioned  as  a  web-enabled  application.  Since  we  wanted  the  software 
aid  to  be  acceptable  to  the  widest  possible  audience,  we  chose  not  to  brand  it  with  our  company 
name;  this  shaped  our  choices  of  where  on  the  web  it  should  be  hosted.  Process  investigations 
began  with  the  Defense  Acquisition  University’s  Defense  Acquisition  Guidebook  (DAG)  and 
were  enhanced  by  two  seminars  presented  to  the  HSIWG  by  Dr.  Dennis  Folds  of  Georgia  Tech 
Research  Institute  (GTRI)  (13)  (14)  and  a  NATO  report  Cognitive  Analysis,  Design  and 
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Evaluation  (COADE)  (15).  Dr.  Folds,  who  described  the  practice  and  analyses  of  HSI  in  the 
context  of  systems  engineering,  appears  at  the  bottom  on  the  right  side  of  figure  8.  Contract- 
funded  research  of  the  COADE  report  revealed  a  process  that  extended  Dr.  Folds’  HSI 
description  to  specifically  include  cognitive  engineering.  This  illustrates  the  advantages  obtained 
from  the  complementary  contract  and  noncontract  work. 

APSE  is  centered  on  acquisition  process  deliverables.  A  list  of  over  300  deliverables  was 
compiled.  This  list  was  pruned  to  arrive  at  the  35  deliverables  included  in  APSE.  Descriptive 
content  was  generated  from  investigations  of  the  systems  engineering,  HSI  and  cognitive 
engineering  activities,  methods  and  software  supports. 

A  deliverable  of  the  SBIR  topic  was  a  toolset.  The  differing  interpretations  between  cognitive 
and  systems  engineering  were  accommodated  by  collecting  methods,  tools  of  applied 
psychology,  as  well  as  identifying  software  aids  that  existed  or  were  required.  Cognitive 
engineering  tools,  tabulated  as  part  of  the  phase  I  effort,  informed  the  APSE  population. 

Software  aids  were  listed  and  investigated  based  on  provider  descriptions  of  features.  We 
planned  to  include  this  list  in  APSE.  However,  we  detennined  this  approach  to  be  inadequate. 
Other  catalogues  of  relevant  software  had  been  made;  these  were  not  found  to  be  useful  for 
people.  The  only  way  to  really  know  if  a  product  suits  a  user’s  purpose  is  to  try  it.  So  we  chose 
to  purchase  and  exercise  select  software  to  demonstrate  how  it  supported  cross-disciplinary 
communications. 

In  the  center  of  figure  8  are  the  APSE-specific  marketing  activities.  Phase  III  marketing  was 
done  for  APSE  in  contrast  to  the  market  development  activities  shown  on  the  right  side  of  the 
figure.  Market  development  encompassed  awareness-raising  and  educational  activities  that  did 
not  feature  APSE.  APSE  was  shown  in  its  various  prototypes  at  meetings,  when  presentations 
were  given  and  at  HSIWG  gatherings  which  are  considered  part  of  our  market-development 
effort.  The  product  was  sometimes  formally  presented  and  sometimes  informally  shown  during 
breaks  or  at  the  end  of  the  day.  We  gathered  use  and  design  inputs  from  potential  users  mostly 
without  soliciting  them. 

Malcolm  Gladwell  (16)  describes  a  type  of  person  who  puts  together  people  with  common 
interests  or  with  solutions  to  one  another’s  needs  as  being  “connectors.”  Emily  Roth  described 
us  as  being  at  the  nexus  of  the  cognitive-systems  engineering  integration  efforts.  This  was  done 
by  participating  as  a  vocal  proponent  at  meetings,  making  mostly  invited  presentations  to  forums 
of  professionals.  These  were  all  market  development  activities,  opportunities  to  educate  people 
on  the  value  of  HSI  and  cognitive  engineering.  In  some  forums,  such  as  the  monthly  Project 
Management  International  (PMI)  meetings,  advocacy  for  systems  engineering  was  required  as 
well. 
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The  HSIWG  brought  together  proponents  of  HSI  and  cognitive  engineering.  It  provided  the 
opportunity  advance  cross-organizational  cooperation  between  the  Human  Factors  and 
Ergonomics  Society  (HFES)  and  IEEE’s  Systems,  Man  and  Cybernetics  Society  (SMCS),  and 
the  American  Society  of  Safety  Engineers.  We  have  already  described  Dr.  Fold’s  participation 
in  HSIWG  meetings.  Contract  funds  also  were  used  to  support  pitstop  design  engineer  Dennis 
Carlson’s  contribution  and  to  purchase  reprints  of  the  special  HSI-themed  edition  of  INCOSE 
Insight  magazine.  One  hundred  copies  of  this  issue  were  distributed  at  Wright-Patterson  AFB 
via  the  Air  Force  Center  for  Systems  Engineering’s  Michael  Mueller,  HSIWG  co-chair.  Four 
hundred  copies  were  distributed  to  exhibitors  at  EITSEC. 

The  APSE  booth  at  I/ITSEC  was  the  culmination  of  the  project.  In  addition  to  demonstrating 
APSE,  passing  around  the  Insight  copies,  and  talking  with  people  interested  in  HSI  and  cognitive 
engineering  at  the  booth,  vendors  were  approached  to  assess  their  awareness  of  HSI.  We  tried  to 
judge  the  participating  training  specialists’  awareness  of  their  role  in  HSI  and  whether  they 
incorporated  HSI  or  cognitive  engineering  as  part  of  their  practice. 

Methods  used  in  executing  the  task  items  in  figure  8  are  described  in  more  detail  below.  First, 
though,  the  assumptions  that  molded  the  approaches  taken  are  discussed. 

3.3.  Assumptions 

As  part  of  APSE  development,  assumptions  associated  with  our  content  were  documented.  It  is 
very  difficult  to  identify  the  ground  on  which  you  stand  while  tread  upon  it.  The  following 
describes  the  position  from  which  we  started. 

First,  we  assumed  the  Air  Force  was  an  advocate  for  embedding  cognition.  The  2004  Air  Force 
Science  Advisory  report  (17)  documented  human  decision  making  and  performance  as  critical  to 
air  combat,  air  mobility  command,  control,  communications,  computers/intelligence 
surveillance,  and  reconnaissance,  and  information  warfare  and  space  operations. 

We  also  assumed  cognitive  engineering  practice  was  sufficiently  mature  to  contribute  positively 
contribute  to  improved  acquisition  outcomes  when  given  the  opportunity  to  participate  on  an 
equal  footing  with  other  specialty  engineering  disciplines,  e.g.,  electrical,  mechanical,  civil, 
reliability,  quality,  etc. 

As  can  be  seen  from  table  2,  the  first  assumption  was  that  tools  existed  and  were  available  to  be 
confederated  into  a  toolset  that  would  align  cognitive  systems  engineering  with  systems 
engineering. 

We  did  not  set  out  to  fundamentally  restructure  systems  engineering  practice  or  the  defense 
acquisition  system.  Some  of  our  cognitive  engineering  colleagues  asserted  that  success  would 
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only  be  achieved  when  the  old  system,  and  current  practitioners,  passed  away.  We  did  not  agree, 
and  did  not  believe  that  was  a  viable  stance.  Customers  depend  upon  project  managers.  Project 
managers  depend  upon  systems  engineers  for  technical  planning,  coordination  and  execution. 
Therefore  managers  and  systems  engineers  have  the  authority  and  resources  to  engage  human 
systems  integrators  and  cognitive  engineers.  Even  with  stories  of  costly  failed  systems  and  user 
inefficiency  piling  up,  we  were  unable  to  conceive  of  an  effective  lever  that  would  result  in  the 
radically  change  of  existing  acquisition  processes.  We  assumed  systems  engineering  success 
could  be  enhanced  by  the  contributions  of  cognitive  engineers. 

We  further  assumed  the  desired  process  for  embedding  cognitive  systems  engineering  in  systems 
engineering  practice  was  based  upon  systems  engineering  fundamentals.  Cognitive 
engineering’s  contribution  to  the  acquisition  life  cycle  was  to  be  based  on  the  process  described 
in  the  DAG. 

Our  phase  I  observations  revealed  there  was,  at  the  time  of  proposal,  no  viable  market  for  our 
product.  Our  plan  was  to  create  open  source  applications,  first  generation  tools,  which  our  target 
users  could  use  in  their  practice  via  the  web.  As  task  8  shows,  we  felt  a  large  part  of  the  effort 
was  going  to  be  educational  -  raising  the  awareness  of  communities,  researchers,  and  acquisition 
practitioners.  At  the  inception  of  our  work,  there  was  little  appreciation  for  cognitive 
engineering  and  little  awareness  of  what  it  was  and  what  value  it  could  deliver.  This  education 
would  take  the  fonn  of  marketing,  shown  in  the  right  half  of  figure  8. 

We  also  believed  that  “not  invented  here”  was  a  substantial  risk  to  the  acceptance  of  our  process 
and  product.  We  observed  rivalries  between  the  services,  between  researchers,  between  the 
disciplines,  between  companies  to  be  THE  ONE  who  solved  the  problem.  We  decided  to  forgo 
credit  for  solving  the  problem  because  we  believed  doing  so  was  essential  to  actually  solving  it. 
Therefore,  we  chose  to  avoid  branding  our  product  wherever  possible.  We  assumed  we  could 
prove  the  value  or  our  outputs  to  potential  users,  and,  once  proved,  they  would  then  use  them  or 
adapt  them  to  their  own  use. 

3.4.  APSE 

Our  approach  involved  exploring  the  differences  between  the  ways  in  which  cognitive  engineers 
preferred  to  work  and  the  ways  in  which  systems  engineers  and  traditional  project  management 
worked.  We  incorporated  this  exploration  into  project  execution. 

At  the  beginning,  the  project  was  run  along  traditional  lines.  We  had  a  firm  schedule  with 
milestones.  Subcontracting  cognitive  engineers  rebelled.  This  was  too  much  of  a  tops  down 
approach  along  the  lines  of,  “This  is  what  the  solution  will  look  like,”  which  was  detennined 
before  any  exploration  had  been  done.  They  felt  this  was  an  embodiment  of  the  system  that  was 
holding  their  practice  in  check.  In  response,  a  less-structured,  collegial,  exploratory  approach 
was  adopted  to  see  how  it  would  work. 
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This  worked  well  at  the  beginning  of  the  project.  It  created  a  more  open,  exploratory  atmosphere 
in  which  we  identified  our  five  target  customers  and  researched  what  they  were  concerned  about 
and  listened  to  their  complaints  about  what  didn’t  work  and  what  needed  to  be  in  place.  APSE 
might’ve  been  completed  earlier  if  we’d  stuck  to  the  original,  tops-down  approach,  but  it 
would’ve  been  a  different  APSE,  one  that  was  less  responsive  to  the  real  problem. 

In  the  final  months  of  the  project,  cognitive  engineers  requested  that  a  schedule  with  milestones 
be  created.  Other  commitments  were  pressing  on  the  subcontract  team.  These  pressures  left 
them  less  free  to  immerse  themselves  in  this  project.  Communications  became  more  sporadic; 
they  became  dissociated  from  the  activities  of  the  prime  contract  team.  The  schedule  structure 
enabled  them  to  plan  APSE  activities  into  their  schedule  which  was  becoming  increasingly 
crowded.  A  schedule  was  put  together,  but  even  then,  it  wasn’t  as  strictly  enforced  as  it 
would’ve  been  for  a  traditional  development  project. 

The  following  subsections  describe  the  approaches  we  took  to  product  and  process  development. 
It  covers  the  boxes  shown  on  the  left-hand  side  of  figure  8  above. 

3.4.1,  Hosting 

It  was  important  that  the  solution  to  the  problem  of  embedding  human  engineering,  cognition  in 
particular,  was  not  seen  as  the  proprietary  work  of  one  company.  While  there  was  a  definite 
desire  among  people  in  the  various  communities  to  solve  the  problem,  to  come  up  with  standard 
processes,  procedures,  and  methodologies  that  could  be  called  upon,  there  was  also  a  competitive 
flavor  to  dealings  as  well.  Companies,  individuals,  branches  of  the  DoD,  even  organizations 
within  the  different  branches,  exhibited  type  A  behavior  -  each  wanted  to  be  the  top  dog  and  to 
have  a  solution  that  favored  their  practice  be  the  one  sought.  “Not  invented  here”  was  a 
component  of  the  social  dynamic  that  had  to  be  recognized  and  accommodated. 

Additionally,  the  prime  contractor  for  this  effort  does  not  supply  cognitive  engineering  services. 
If  we  were  to  have  found  THE  solution  to  the  problem,  it  would’ve  put  the  experts  in  the 
cognitive  engineering  field  in  a  difficult  position. 

For  these  reasons,  we  sought  to  achieve  a  community  solution.  First,  we  worked  to  form 
collaborations.  Between  phase  I  and  phase  II  a  team  of  six  cognitive  engineering  and  two 
systems  engineering  companies  crafted  a  proposal  for  the  Office  of  Naval  Research  to 
collaboratively  extend  the  work  we’d  been  doing  in  this  area.  The  proposal  was  not  funded,  but 
it  did  provide  a  forum  in  which  ideas  were  shared.  Peoples’  understandings  of  the  problem 
changed. 

Once  phase  II  awards  were  announced,  a  non-disclosure  agreement  was  executed  with  one  of  the 
other  awardees.  We  each  sought  to  sculpt  our  work  plan  so  that  it  in  combination  they  would 
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provide  greater  value  to  the  Air  Force.  These  efforts  were  unsuccessful  primarily  due  to 
differing  levels  of  maturity  in  our  plans.  Our  plan  was  highly  structured  which  seemed  to  restrict 
their  flexibility;  their  plan  was  more  nebulous  which  made  finding  a  fit  difficult. 

Our  third  approach  was  to  take  our  name  off  the  product.  This  was  a  difficult  business  decision. 
This  being  our  first  contract  of  substance,  we  were  hoping  it  would  become  a  springboard  to 
future  business.  Adopting  a  posture  of  anonymity  did  not  support  those  hopes.  Nevertheless, 
this  seemed  to  be  the  best  value  approach  for  the  Air  Force,  so  it  was  adopted.  It  turned  out  to  be 
more  difficult  than  anticipated;  the  very  act  of  seeking  feedback  or  exchanging  infonnation  made 
it  our  solution  rather  than  the  community  solution  we  sought. 

APSE  was  planned  to  be  a  web-enabled  application.  A  hosting  plan  that  help  people  to  regard 
APSE  as  a  community  solution  was  sought.  We  subcontracted  with  Wright  State  University’s 
psychology  department.  They  would  perfonn  research  and  host  the  product  on  their  web  site 
when  it  was  completed.  Unfortunately,  this  arrangement  did  not  work  out  in  the  long  term. 

Instead,  we  set  out  to  purchase  the  URL  “apse.com.”  When  it  turned  out  to  be  unavailable,  we 
instead  purchased  the  URL  “acprac.com,”  and  hosted  APSE  there  without  branding.  It  remains 
to  be  seen  if  this  approach  will  be  effective.  While  it  does  address  the  not  invented  here 
dynamic,  it  lacks  the  validity  of  a  known,  trusted  entity.  We  wonder  if  people  will  be  suspicious 
of  the  content  and  so  choose  not  to  use  APSE.  The  application  has  been  crafted  to  capture  traffic 
statistics,  so  we  will  be  able  to  tell  if  people  do  engage  and  to  what  extent  they  explore  the 
material. 

3.4.2.  Model  Process 

Phase  I  required  “a  model  process  for  cognitive  systems  engineering  and  improvements  that 
would  be  needed  to  make  the  process  attractive  to  government  acquisition  managers  and 
industry.”  Phase  II  required  “executable  software  tools  that  instantiate  the  model  process  in  the 
context  of  an  extension  to  classical  systems  engineering.” 

At  the  HSIWG  meeting  at  2007  INCOSE  International  Workshops,  Dr.  Jennifer  Narkevicius 
pointed  out  that  HSIWG  didn’t  have  the  authority  to  modify  documents  specifying  acquisition 
processes  and  procedures.  There  followed  a  discussion  about  the  difficulty  of  making  such 
changes,  the  lengthy  authoring,  review  and  approval  cycle,  and  all  of  the  lower  level  government 
and  industry  processes  and  tasks  that  would  be  affected.  Additionally,  when  the  topic  of  process 
was  raised  among  systems  engineers,  a  lengthy  debate  was  spawned  about  process  models.  The 
waterfall  process  was  rejected.  Incremental  processes  were  preferred.  Some  advocated  spirals. 
Some  advocated  fountain-  or  star-shaped  models. 
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Since  our  team  set  as  a  goal  to  retire  the  problem  in  a  constrained  period  of  time,  this  situation 
was  discouraging.  It  led  us  to  consider  whether  the  changes  to  the  acquisition  process  needed  to 
happen  in  order  to  instantiate  the  required  changes.  As  if  to  confirm  our  supposition,  Robert 
Machol,  in  SYSTEMS  ENGINEERING  HANDBOOK  (17)wrote, 

“ The  steps  of  system  design  are  logical  steps,  but  they  are  not  performed  in  order.  Logically, 
one  must  formulate  the  problem  before  one  solves  it;  in  fact,  one  does  both  simultaneously 
through  the  system-design  process.  Because  the  problem  cannot  be  adequately  formulated 
until  it  is  well  understood,  and  because  it  cannot  be  well  understood  until  it  has  been  more  or 
less  solved,  the  two  are  inseparable.  ” 

We  explored  the  DAG  and  systems  engineering  processes  documented  in  the  INCOSE  Systems 
Engineering  Handbook  (2),  the  IEEE  Computer  Society’s  1220  Standard  for  Application  and 
Management  of  the  Systems  Engineering  Process  (3)  and  the  Lockheed  Martin  Space  Systems 
Division  Systems  Engineering  Manual  (18).  The  user-centered  spiral  developed  by  Dr.  Robert 
Hoffman  (19)was  also  considered. 

From  these  investigations,  we  concluded  process  models  might  vary,  but  the  in-process 
deliverables  were  invariant.  Whether  a  waterfall  or  spiral  was  implemented,  user  requirements 
still  needed  to  be  gathered;  functions  had  to  be  allocated;  specifications  and  verification  plans 
had  to  be  generated;  reviews  had  to  be  conducted.  This  finding  was  attractive  because  it 
removed  the  burden  of  process  change.  If  the  most  important  in-process  deliverables  could  be 
identified,  those  that  had  the  greatest  leverage  for  embedding  HSI  and  cognitive  engineering  in 
systems  engineering,  then  they  could  be  mapped  to  any  process.  When  the  deliverables  were 
generated,  how  many  times  and  how  often  would  differ,  but  disciplinary  contributions  would  be 
the  same  no  matter  what  the  process.  As  this  was  an  Air  Force  project,  a  combined  JCIDS  and 
DAS  process  was  selected  as  the  process  to  which  deliverables  would  be  mapped. 

Over  300  deliverables  were  identified  from  our  review  of  process  documentation.  The  list  is 
provided  in  Appendix  A.  This  needed  to  be  culled  to  the  most  influential  for  our  purposes. 
Cognitive  engineering  contributions  to  the  deliverables  were  documented.  A  “top  ten”  list  of 
most  valuable  cognitive  engineering  activities  was  compiled  based  the  experiences  of 
practitioners.  The  contained  those  activities  they  had  been  called  upon  most  frequently  to 
perfonn  and  which  had  had  the  most  influence  on  the  mission  perfonnance.  Appendix  B 
contains  the  list.  Table  3  is  an  excerpt. 
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Table  3:  Excerpt  from  list  of  most  influential  cognitive  engineering  activities 


Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

Field  Studies 

Cognitive  Task  Analysis 
Critical  Decision  Method 
Ethnography 

Surveys 

Questionnaires 

Interviews 

Cognitive  Task  Analysis 
Scenarios 

Environment  Characteristics 
SKAs 

Team  Dynamics 

Early  Operational  Assessment 
Operational  Assessment 

Operational  Testing 

Operational  Test  Agency  Report 
ofOT&E  Results 

Sustainment  Assessments 
Post-Deployment  Reviews 

Data  Asset  Identification 

User  Requirements 

Functional  Requirements 

E1CI  Design  Specifications 
TES/TEMP 

Developmental  Test  and 

Evaluation 

Live  Fire  Test  and  Evaluation 

Test  Events 

Product  Support  Plan 

Training  Plan 

Beyond  LRIP  Report 

Full  Rate  Production  Decision 
Review 

User  Reviews 

Programmatic  Environment 

Safety  and  Occupational  Health 
Evaluation  (PESHE) 

Support  Strategy 

The  other  eight  integration  points  are 

2.  User  Scenarios 

3.  Walkthroughs 

4.  Task  Analysis 

5.  User  Profiles 

6.  CONOPS  -  Navigation  Model  (high-level  view  allows  navigation  through  system) 

7.  CONOPS  -  Concept  Model  (development  around  system’s  central  concept) 

8.  Training  Requirements 

9.  Feature  Definition 

The  activities  are  points  of  integration  into  the  systems  engineering  process.  The  table  associates 
methods  and  intermediate  products.  In  the  final  column,  deliverables  from  the  list  of  300  are 
associated  with  the  integration  point  as  shown  in  the  second  column  of  figure  9  which  illustrates 
the  process  used  to  reduce  the  300  deliverables  to  the  final  35  that  were  incorporated  in  APSE. 
Acronyms  used  as  abbreviations  in  column  2  are  expanded  in  the  List  of  Symbols,  Abbreviations 
and  Acronyms  in  the  back  sections  of  this  report. 
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Top  Ten 
CogEng 
Activities 


Phased 

Acquisition 

Activities 


Binned 

Process 

Topics 


Selected 

35 

Deliverables 


•Field  Studies 

JCIDS 

•Operations 

•HSI  Activities 

•UserScenarios 

•FAA 

Concepts 

•SE  Activities 

•Walkthroughs 

•FNA 

•  Measures  of 

•CogEng  Activities 

•Task  Analysis 

•FSA 

Goodness 

•Management 

•User  Profiles 

•DOTMLPF  Analysis 

•Function  Analysis  & 

Activities 

•CONOPS  Navigation 

•IMA 

Allocation 

Model 

•AMA 

•  DOTMLPF  Analyses 

•CONOPS  Concept 

•PIA 

•  Project  Activity 

Model 

•ICD 

Planning 

•Training 

•Operational  and  Support 

•Capability  Concepts 

Requirements 

Attributes  of  system 

•  Requirements  and 

•  Feature  Definition 

•Net-Ready  KPP 

Constraints 

•  HSI  Approach 

•  Manpower 

•Embedded  Systems 

Estimation  and  Jobs 

Approach 

•Modeling& 

•RiskAnalysis 

Simulation 

•KPPThresholds 

•  Design  & 

•CDD 

Specification 

•CPD 

•  Information  Flow  & 

Concept  Refinement 

Use 

•SEP with  HSI  Plan 

•  Economic  Analyses 

•  IMP/IMS 

•Operational 

•Data  Management  Plan 

Evaluation 

•ISP 

•Modeling and  Simulation 
•User  Needs 

•Operations&  Sustainment 
Requirements 
•Capability  Analysis 
•Environmental  Constraints 
•Performance  & 
Verification  Objectives 
•Function  Analysis 
•Define  Components 
•Component  Requirements 
•ConceptVerification  vs. 
User  Needs 

•Concept  Dis-/Advantages 
•Select  Concept 
•  Preliminary  Specification 
•Technology  Development 
Strategy 
•AoA  Review 
•Initial  Technical  Review 
Technology  Development 
&  Demonstration 


Figure  9:  Reducing  from  300  to  35  APSE  deliverables 
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Themes  were  identified  from  the  phased  list  which  allowed  us  to  identify  process  topics  to  be 
addressed  in  APSE.  The  third  column  shows  the  13  categories  that  evolved.  Fifty-six 
in-process  deliverables  were  identified  as  the  most  relevant  to  these  categories.  Duplicates  were 
identified  which  reduced  the  number  to  50.  HSI,  cognitive  engineering,  systems  engineering  and 
project  management  activities  were  mapped  into  the  categories.  This  enabled  us  to  prioritize  the 
50  remaining  deliverables,  and  place  them  in  the  appropriate  phases. 

We  have  described  how  the  cognitive  engineering  activities  were  arrived  at.  Systems 
engineering  activities  were  taken  from  manuals  and  handbooks  and  the  author’s  experience. 
Project  management  activities  are  described  in  the  DAG.  The  HSI  activities  were  taken  from  a 
presentations  covering  HSI  analysis  given  by  GTRI’s  Dr.  Dennis  Folds. 

Dr.  Dennis  Folds  of  Georgia  Tech  Research  Institute  presented  “Human  Systems  Integration”  at 
the  June,  2007  HSIWG  meeting.  Figure  10  is  a  slide  from  that  presentation.  Dr.  Folds 
subsequently  presented  a  half-day  seminar  on  HSI  Analysis  at  the  January,  2008  HSIWG 
meeting.  Those  slides  are  downloadable  from  within  APSE. 

Mission  Task  Analysis  in  Design 


Figure  10:  HSI  Analyses  per  Dr.  Dennis  Folds  (13) 

Shortly  after  Dr.  Folds’  presentation,  we  reviewed  the  NATO  COADE  (15)  report.  The  COADE 
study  offered  the  model  process  for  cognitive  engineering  analysis  and  design  that  is  depicted  in 
figure  11.  Note  that  the  figure- 10  mission  analysis,  function  analysis,  and  task  analysis  matches 
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those  in  the  first  line  of  the  COADE  process.  Dr.  Folds’  description  of  task  analysis  matches 
with  the  COADE  behavior  task  analysis. 

By  comparing  figures  10  and  1 1,  it  can  be  seen  that  HSI  analyses  provide  inputs  into  cognitive 
engineering  analyses.  Since  cognitive  engineering  is  a  component  of  HSI,  some  of  the  HSI 
analyses  are  actually  perfonned  or  supported  by  cognitive  engineering  methods.  For  example, 
cognitive  engineering  supports  HSI  manning  estimates,  infonnation  requirements  definition,  and 
error  prediction. 

Storyboards  were  constructed  to  connect  the  process  and  products  of  JCIDS  and  the  DAS  with 
those  of  the  HSI  and  COADE  analysis  processes  and  with  the  activities  that  were  documented  in 
the  phase  I  flowcharts.  In  doing  so,  we  confirmed  that  HSI  and  cognitive  engineering  could  be 
included  in  the  combined  JCIDS/DAS  process  without  requiring  its  revision.  In  addition,  this 
allowed  us  to  complete  the  process  shown  in  figure  9  which  reduced  the  50  deliverables  to  the  35 
included  in  APSE. 
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Figure  1 1 :  COADE  Model  of  Cognitively-centered  System  Design 
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3.4.3.  Interface 

For  some  time  process  analysis,  deliverable  definition  and  content  generation  subsumed  the 
entire  effort.  The  cognitive  engineering  members  of  our  team  began  to  have  trouble  conceiving 
the  evolving  product  and  insisted  that  some  attention  be  paid  to  the  user  interface.  A  series  of 
prototypes  was  generated.  Infonnal  feedback  was  provided  by  potential  users  including 
members  of  the  design  team.  The  evolution  of  the  APSE  main  page  is  provided  in  Appendix  C. 

The  first  version  of  APSE  was  created  using  the  Backpack  web  2.0  tool  by  37  signals  (20).  It 
was  primarily  repository  for  content  at  the  time  when  the  Backpack  deliverables  list  contained 
150  items.  Backpack  poorly  aggregated  related  content.  Descriptions  were  separated  from 
supporting  graphics,  documents  and  files.  It  was  difficult  to  edit  and  format  text.  Each  page  had 
separate  pennissions  which  were  difficult  to  manage.  Backpack  did  have  collaboration  tools 
such  as  texting  and  chat  that  would’ve  been  advantageous  if  APSE  had  become  the  collaborative 
design  tool  it  was  originally  envisioned  to  be. 

It  seemed  wise  to  use  a  cognitive  engineering  product  to  communicate  the  benefits  of  the 
discipline.  A  concept  map  version  of  the  main  page  was  devised.  It  is  shown  in  figure  12. 

Many  of  the  functions  of  the  final  product  can  be  seen  in  this  view.  The  blue  ovals  contain  the 
featured  content  -  deliverables,  methods  and  software.  At  that  time,  we  considered  incorporating 
high  value  activities  as  one  of  the  functions.  Once  we  slimmed  down  to  35  deliverables, 
however,  all  the  content  was  considered  to  be  high  value  and  this  feature  was  dropped. 

The  concept  map  version  was  not  favorably  reviewed.  Surprisingly,  the  negative  comments 
came  from  a  proponent  of  concept  maps.  This  person  felt  the  map  was  useful,  but  was  not 
appropriate  for  the  main  page.  It  did  not  lead  users  through  the  material  and  made  action  steps 
difficult  to  discern.  Enhancements  in  the  fonn  of  process  bars  and  links  to  high  value  activities 
were  added,  but  the  design  was  not  acceptable.  Subsequently,  an  input/output  version  and  a 
tabbed  version  were  developed.  The  design  team  didn’t  feel  these  provided  the  desired  look 
either. 
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Are  ways  to  accomplish 


Figure  12:  Concept  Map  for  APSE  Front  Page 

Knowing  that  APSE  would  be  a  “tough  sell,”  a  dynamic,  attractive,  fun-to-use  interface  was 
sought.  We  looked  for  novelty  that  would  attract  users  along  with  functionality  that  would  be 
useable  and  useful.  “Best  of  breed”  web  interfaces  were  explored.  These  were  found  by 
perfonning  internet  searches  on  award  winning  web  sites.  Web  sites  of  museums,  which  were 
attractive  and  fun,  were  favored.  Two  dynamic  and  attractive  sites  were 
http://www.brainpop.com/,  and  http://xplane.eom/#/problems/.  They  contained  a  lot  of  content 
and  were  engaging,  and  served  to  set  our  expectations  for  an  acceptable  APSE  design. 

We  created  storyboards  of  potential  main  and  deliverables  pages  using  Microsoft  PowerPoint. 
Cognitive  walkthroughs  of  the  storyboards  were  conducted.  These  were  used  to  assure  that  the 
interface  would  guide  users  to  execute  the  actions  we  wanted  them  to.  For  example,  a  win  for  us 
if  the  project  manager  took  away  the  following:  1)  When  to  include  cognitive  engineers  and  HSI 
personnel  in  planning;  2)  How  to  discriminate  between  subcontractor  offerings;  3)  An 
understanding  of  what  contributions  to  expect  from  cognitive  engineering  and  HSI  practitioners. 
After  internal  discussion,  a  mockup  was  shown  to  Dr.  Fran  Greene  of  the  Air  Force  HSI  Office. 
Dr.  Greene  mentally  translated  our  message  and  described  a  new  interface  which  is  shown  in 
figure  13. 
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!  APSE  Home  -  Internet  Explorer  provided  by  Dell 
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APSE  supports  the  day-to-day  work  of  people  involved  in  a  product's  life  cycle. 
It  helps  developers  carry  out  the  tasks  that  have  the  greatest  impact  on  user 
performance  and  total  cost.  APSE  identifies  methods  and  software  aids  that 
help  system  developers  to  get  the  job  done. 

How  do  I  plan? 


Product 

Gov't 

Program 

Manager 

Contract 

Program 

Manager 

Systems 

Engineer 

HSI 

Analyst 

Cognitive 

Engineer 


Mouse  Over 

Initial 

Capabilities 

Document 


MouseOver 

Request 

For 

Proposal 


X 


X 

X 

X 


To  Do 


What  do  I  do? 

I 


Product  Pull  Down 


Key  Tasks 

♦PM 

♦SE 

•HSI 

•CE 

Manhours 

♦PM 

•SE 

♦HSI 

♦CE 

Methods  and  Procedures 

♦PM 

♦SE 

•HSI 

•CE 

Software  Aids 

♦PM 

♦SE 

•HSI 

•CE  ‘ _ 


JCIDS:  Customer  requirements  collected. 
Customer  need  is  identified. 


ro 

E 

E 

rs 

GO 


Multiple  rough  concepts  generated. 
Technical,  human  and  cost  implications 
explored. 

Technical  and  human  requirements 
stated  and  documented. 


U 

"O 

O 


re¬ 

s¬ 

ets 

E 

E 


Product  summary  text. 


D_  GO 


Our  Story 


What  will  give  me  highest  return  on  assets  employed? 
Figure  13:  APSE  main  page  mockup  post  Dr.  Greene  feedback 
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The  panels  in  the  center  reflect  Dr.  Greene’s  descriptions.  On  the  left  is  “what  do  I  do?”  On  the 
right  is  “how  do  I  do  it?”  The  design  is  still  static  and  not  very  interesting,  but  the  functionality 
is  very  close  to  that  of  the  final  product. 

Discussions  among  the  design  team  revealed  a  shortfall  in  the  cognitive  engineering  process  that 
is  shown  in  figure  14.  Engineers,  in  the  applied  science  process,  have  a  systems-design- 
production/manufacture  progression.  Interpretation  between  the  production  function  and  design 
function  is  provided  by  the  core  systems  engineering  team.  Cognitive  engineers,  in  the  applied 
psychology  process,  lack  the  design  function.  In  the  absence  of  cognitive  engineering 
participation,  user  interfaces  are  developed  by  software  engineers  or  web  developers  who  are  not 
trained  or  equipped  to  develop  intuitive  and  effective  interfaces,  particularly  for  mission  critical 
functions.  A  specialist  in  human-computer  interface  (HCI)  design  is  sometimes  called  upon  to 
fill  this  void,  but  the  role  is  not  institutionalized.  For  commercial  web  interfaces,  that  function  is 
filled  by  a  graphic  design.  Subsequent  to  this  conversation,  a  graphic  designer  was  added  to  our 
team.  The  designer,  working  from  figure  13,  created  a  flowing,  artistic  theme. 


Figure  14:  Design  function  missing 


After  the  visual  design  was  complete,  it  was  turned  over  to  a  software  engineer  who  added 
dynamic  windowing  and  graphics.  This  resulted  in  the  desired  interesting  and  functional  design 
we  had  been  seeking. 

3.4.4.  Content 

The  concept  of  a  loose  confederation  of  analysis  tools  was  proposed.  APSE  wound  up  as  a 
planning  tool  with  many  of  the  elements  of  an  e-leaming  system.  How  did  this  transformation 
occur? 

First  we  noted  that  the  software  needed  to  instantiate  the  phase  I  model  process.  Figure  15 
aggregates  the  attributes  of  that  process.  The  content  in  figure  15  does  not  describe  a 
confederation  of  analysis  tools.  It  looks  like  a  project  planning  and  management  tool. 
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■  APSE  Content  (preliminary) 

□  Project  Activities 

□  Project  Authority 

□  Project  Definitions  and  Glossary 

□  Project  Design 

□  Project  Environment 

□  Project  Estimating 

□  Project  Mission  Statement 

□  Project  Organization 

□  Project  Resources 

□  Project  Responsibilities 

□  Project  Requirements 

□  Project  Risk 

□  Project  Roles 

□  Project  Schedule 

□  Project  Stakeholders 

a  Regulations  and  Standards 

APSE  Action:  Answers  feed  into  initial  (freeware)  project  database 


Figure  15:  Activities  flow-charted  in  phase  I  provided  initial  content  for  APSE  prototype 

Next,  we  found  out  more  about  the  disposition  of  the  US  Navy’s  Human  Centered  Design 
Environment  (HCDE)  created  which  had  been  explored  in  phase  I.  While  the  NAVSEA  has  not 
maintained  use  of  the  HCDE  in  the  Navy  Capability  Development  Process,  it  has  retained 
Interchange  SE.  The  demise  of  HCDE  gave  us  pause.  NAVSEA  had  developed  a  fairly  capable 
analysis  capability  which,  by  reports,  still  some  needed  work  and  then  ceased  to  fund  it. 

Finally,  there  was  the  question  of  users.  Systems  engineers  at  HSIWG  meetings  were  asking, 
“What  is  cognitive  engineering  anyway?”  “Where  can  I  find  out  more?”  Only  a  small  subset  of 
the  systems  engineers  knew  what  HSI  was.  At  PMI  chapter  meetings,  the  author  questioned 
attendees  to  determine  their  awareness  of  HSI  and  found  none,  so  it  was  not  on  the  management 
radar  screen  either. 

At  the  Integrated  Design  and  Process  Technology  9th  World  Conference  in  June,  self-styled 
systematist  Jack  Ring  said  off-handedly,  “Systems  Engineering  artifacts  are  e-learning  products 
for  Engineering.”  This  led  to  modification  of  the  APSE  concept.  They  were  expressed  as  the 
following  postulates,  corollaries  and  elaborations. 

•  Postulate  1 :  APSE  is  essentially  an  e-Leaming  tool. 

o  Structure  APSE  around  life-cycle  deliverables  taken  from  systems  engineering 
manuals  and  the  DAG. 
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o  Appears  that  Systems  Engineering  Plan  (SEP)  may  provide  a  backbone  for  the  e- 
leaming  environment. 

•  Postulate  2:  APSE  must  provide  valuable  functionality  that  draws  seasoned  users  into  the  e- 
leaming  experience. 

•  Corollary  1 :  APSE  must  not  require  user  activity  that  is  not  perceived  as  value-added  that 
would  be  routinely  by-passed. 

o  Provide  users  with  new  material, 
o  Provide  users  with  reference  material, 
o  Provide  users  with  software  aids. 

•  Postulate  3:  Organizations,  the  various  systems  engineering  standards,  senior  practitioners 
each  have  their  own  process  model  consisting  of  the  logical  systems  design  steps. 

•  Postulate  4:  It  is  not  so  important  to  instantiate  a  process  but  to  support  the  activities  that  go 
into  executing  the  logical  steps. 

•  Postulate  5:  APSE  will  support  key  activities  that  need  to  be  accomplished  in  a  system’s  life 
and  help  users  to  understand  when  it  is  appropriate  to  execute  those  activities. 

•  Corollary  2:  APSE  will  not  attempt  to  define  a  single,  definitive  process. 

o  APSE  is  a  structure  of  supported  activities  identified  in  phase  I  centered  on  the 
products  (hardware,  software,  planning,  design,  sustainment  artifacts)  that  must  be 
generated  for  a  system  during  its  lifecycle. 

o  However,  for  novices,  an  exemplary  framework  is  required  in  order  to  put  the  APSE 
deliverables  in  content. 

The  architecture  was  refocused  from  being  primarily  a  loose  confederation  of  software  to  a  work 
aid  that  had  a  deliverable  focus.  Details  for  deliverable  generation  were  co-located  within  a  web- 
enabled  software  application.  Infonnation  about  deliverable  preparation  was  broken  into  several 
pages  —  a  summary  page  and  a  detail  page. 

At  the  time  the  postulates  were  fonnulated,  the  software  confederations  had  not  been  abandoned. 
A  freeware  version  of  the  confederation,  called  a  Simple,  or  Study,  Project,  would  be  available 
by  means  of  the  APSE  application.  Recommendations  would  be  made  for  a  Professional  Project 
analysis  architecture  which  would  require  separate  purchases  of  contributing  software  by  the 
organization  or  individual  using  APSE.  Figure  16  lists  the  features  of  the  revised  APSE  concept. 
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■  Plan  a  project 

□  e-Learning  tool  to  guide  managers,  project  planners  and  students  through  setting  up  a 
project. 

■  Start  a  simple  project 

□  Users  employ  the  freeware  version  of  APSE  in  conjunction  with  "Plan  a  Project"  to 
initiate  a  project. 

■  Start  a  professional  project 

□  Users  are  provided  guides  in  selection  and  setting  up  a  project  incorporating 
professional-caliber  aids. 

□  Work  captured  in  the  "Simple  Project"  environment  is  subsumed  by  the  professional 
database. 

■  Work  on  existing  project 

□  Log  in  and  continue  work  on  an  existing  project. 

■  Manage  project  workspace 

□  Global  APSE  administration  function 

■  Manage  personal  workspace 

□  Allows  users  to  tailor  APSE  to  their  personal  preferences 

Figure  16:  Features  of  APSE  e-learning  concept 

The  workspaces  were  subsequently  eliminated  from  the  concept  due  to  software  considerations 
that  will  be  described  in  the  Software  section,  below. 

There  are  many  handbooks,  guidebooks,  standards  and  policies  that  guide  acquisition  practice. 
For  example,  the  DAG  lays  out  the  entire  acquisition  process.  It  is  a  massive  and  impressive 
work.  Why  would  people  go  to  APSE  instead?  Leaning  on  recent  work  performed  for  DARPA 
on  the  topic  of  rapid  and  accurate  information  transfer,  we  found  that  these  massive,  impressive 
handbooks  are  rarely  accessed.  During  our  first  time  through  the  DAG,  typographical  errors  and 
dropped  links  were  caught,  listed  and  sent  to  the  DAG  webmaster.  Fie  was  delighted  to  get  the 
updates,  but  even  more  to  hear  that  someone  had  actually  used  the  document! 

We  find  that  people  don’t  have  time  to  read  the  complete,  detailed  documents  that  describe  in 
excruciating  detail.  They  are  looking  for  infonnation  they  need  just-in-time.  The  internet  has  set 
their  expectations.  They  can  perform  a  search  and  have  information  needed  instantaneously. 
True,  they  could  upload  a  standards  document  and  search  for  a  desired  topic,  but  even  then,  it  is 
not  presented  in  actionable  for. 

Standards  and  manuals  are  generally  at  too  high  a  level  for  actual  practice.  The  Lockheed  SSD 
we  reviewed  is  an  exception  to  that.  It  is  a  cookbook  for  the  systems  engineering  process 
providing  step-by-step  descriptions  of  activities  as  well  as  the  forms  required  to  complete  tasks. 
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APSE  is  designed  to  provide  the  just-in-time  information  people  require.  Not  only  that,  it  is 
sparsely  populated.  By  that  we  mean  that  few  words  are  used  to  delimit  the  deliverable  being 
treated.  Text  fields  were  purposely  kept  short.  The  temptation  to  shrink  white  space  in  the 
graphic  design  was  ignored.  Our  goal  was  to  make  it  possible  to  obtain  a  conversational 
understanding  at  a  glance.  The  architecture  then  provided  succinct  materials  for  going  deeper 
into  the  subject  as  required  to  satisfy  the  APSE  user’s  need. 

APSE  does  not  dismiss,  replace  or  replicate  existing  information  repositories.  Hyperlinks  to 
relevant  DAG  pages  and  detailed  descriptions  of  cognitive  engineering  methods  on  the  MITRE 
Mental  Models  web  site  (21),  for  example,  were  incorporated  into  APSE.  When  representative 
support  software  is  listed,  links  to  the  vendor  web  sites  were  included. 

Templates  that  directly  help  a  person  to  complete  a  specific  deliverable  or  support  product  were 
also  included.  These  are  another  example  of  the  just-in-time  information  provided  by  APSE. 
Table  4  provides  an  example  of  the  Decision  Requirements  Table  developed  by  Klein  et  al.  (22) 
that  is  included  with  the  Task  Analysis  deliverable.  The  document  is  provided  in  downloadable 
Microsoft  Word  format  so  users  don’t  even  need  to  create  it  for  themselves. 


Table  4:  Klein  et  al.  Decision  Requirements  Table  Example 


What  is  the 
difficult  decision? 

Why  is  it  difficult? 

How  is  the 
decision  made? 

Recommend 
Human 
Computer 
Interface  aid. 
Identify  how  will 
it  help? 

Decision 

Frequency 

Number 

of 

Decision 

Incidents 

Discriminating 
vehicles  from 
tracks. 

Operator  must 
discriminate  non- 
combatants  from 
adversaries  before 
making  engage 
recommendation. 

Operator  relies  on 
software 
translation  of 
hyperspectral 
imagery  to  match 
sensor  returns  to  a 
priori  vehicle 
types. 

Improve  a  priori 

typing  and 

provide 

correlation 

certainty 

(ambiguity) 

indicator. 

12 

7 

As  a  final  note  on  content,  APSE  should  ideally  be  a  collaborative  tool  that  receives 
contributions,  updates  and  fosters  debate  within  the  community.  We  have,  for  the  moment, 
abandoned  the  notion  of  APSE  as  a  tool  to  which  community  members  can  contribute.  After 
some  exploration,  we  found  no  way  incentivize  users  to  contribute  content  to  APSE.  We  could 
see  people  loading  research  papers  and  general  infonnation  into  a  repository,  but  details  of  their 
day-to-day  practice  are  proprietarily  guarded.  Extensive  uploads  of  generation  information 
would  have  created  a  daunting  infonnation  repository  with  all  of  the  detriments  of  and  none  of 
the  benefits  of  the  DAG’s  organization. 
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3.4.4.I.  Deliverables 


APSE  content  was  centered  on  deliverables  and  descriptions  of  the  tasks,  manhours,  methods 
and  software  aids  that  support  deliverable  development.  We  have  not  included  all  in-process 
deliverables  in  APSE.  Section  3.4.2  describes  the  process  that  was  used  to  select  the  35 
deliverables  that  are  the  centerpiece  of  APSE.  Essentially  deliverables  were  selected  because  of 
their  value  in  moving  cognitive  engineering  into  the  process  and  cognitive  attributes  into 
systems. 

The  generated  list  of  deliverables  was  biased  toward  technical  products,  such  as  requirements, 
simulations  and  analyses.  That  is  because  these  were  the  products  that  cognitive  engineers  have 
historically  been  asked  to  contribute.  This  bias  has  limited  their  practice.  For  example,  Dr. 
Kelly  Neville  pointed  out  during  phase  I  discussions  that  cognitive  engineers  were  brought  on 
late  in  the  development  cycle.  When  changes  to  user  interfaces  were  recommended,  software 
engineers  pushed  back  claiming  this  to  be  requirements  creep.  Requirements  creep  is  always  a 
bad  thing.  It  is  costly.  It  is  disruptive.  Dr.  Neville  felt  her  contributions  were  at  the  mercy  of 
software  engineers  and  generally  were  not  adopted. 

Systems  engineers  recognize  that  requirements  do  change  during  development.  True,  it  becomes 
more  difficult  to  do  this  as  the  product  matures  because  of  cascading  effects  to  interrelated 
subsystems,  but  systems  engineers,  as  a  matter  of  course,  adjust  requirements  when  perfonnance 
shortfalls  are  found  in  test  in  order  to  rebalance  the  system.  Rebalancing  assures  that  the 
delivered  system  still  meets  mission  requirements. 

There  is  a  process  for  requirements  change  of  which  Dr.  Neville  was  not  aware,  that  is  the 
configuration  management  process  executed  by  the  configuration  control  board  (CCB).  It  is  a 
technical  management  process  that  was  left  off  the  original,  reduced  set  of  deliverables.  Many 
technical  management  deliverables  replaced  purely  technical  deliverables  in  the  set  because  they 
were  enablers  for  cognitive  engineering  practice. 

3.4.4.2.  Tools 

Lack  of  a  common  lexicon  is  one  of  the  barriers  to  cognitive  engineering  and  human  systems 
integration  practice.  The  word  “tools”  encompasses  a  topic  on  which  a  systems  engineer  and 
cognitive  engineer  can  have  a  lengthy  discussion  without  ever  realizing  that  they  are  speaking  of 
two  entirely  different  things. 

The  HSIWG  identified  this  challenge  during  its  formative  days  (23).  There  are  three  areas  where 
confusion  arises.  First,  the  same  word  can  be  used  to  mean  two  different  things.  Second, 
different  words  can  be  used  to  name  the  same  thing.  Third,  people  are  unfamiliar  with  tenns 
used  by  practitioners  of  other  disciplines;  this  becomes  especially  damaging  to  integrated 
practice  when  people  don’t  seek  clarification. 
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Many  cognitive  engineering  practitioners  come  from  research  backgrounds  where  tools  refer  to 
questionnaires  or  analysis  methods;  their  training  does  not  customarily  include  computer 
programming.  On  the  other  hand,  engineers  cut  their  teeth  on  programming.  Computers  and 
software  that  run  then  are  their  tools.  The  question  of  whether  cognitive  engineers  should  be 
trained  to  write  software  and  develop  user  interfaces  was  raised  several  times  during  the  period 
of  performance. 

3.4.4.2.1,  Methods 

Identifying  related  methods  was  one  of  the  tasks  of  deliverable  content  population.  This  is 
important  because  it  addresses  the  third  lexical  challenge  -  practitioners  from  other  disciplines 
who  do  not  know  the  meaning  of  terms  used  by  others.  Systems  engineers  were  observed 
closely  at  one  of  the  INCOSE  symposia.  Instances  when  new  terminology  was  introduced  were 
particularly  noted.  Those  observed  who  found  themselves  faced  within  unfamiliar  tenninology 
preferred  to  bluff  their  way  through  the  conversation  rather  than  admit  they  didn’t  know.  The 
author  observed  this  frequently  when  attending  classes  at  MIT.  Very  intelligent,  competitive 
students  didn’t  want  to  show  weakness  by  admitting  ignorance.  Questions  were  rarely  asked  in 
public. 

Systems  engineers  and  project  managers  are  unfamiliar  with  cognitive  engineering  and  HSI 
methods.  Sometimes  they  may  have  heard  the  term,  but  not  had  the  time  to  investigate  its 
meaning.  Or  perhaps  the  tenn  wasn’t  relevant  to  the  work  they  were  currently  doing.  This  leads 
to  situations  that  cognitive  engineers  experienced  on  DDX.  The  systems  engineers  welcomed 
them,  said  they  could  really  use  the  cognitive  engineer’s  help,  and  next  asked,  “What  do  you 
do?” 

Project  plans,  sections  of  requests  for  proposal  and  other  guidance  documents  require  that 
interacting  practitioners  understand  the  tenninology  of  the  HSI  and  cognitive  engineering.  If 
authors  of  these  documents  are  unaware  of  the  meaning  of  the  terms,  confusing  or  misleading 
text  is  the  result.  When  reviewers,  such  as  a  project  manager  see  things  they  don’t  understand  on 
the  integrated  master  plan,  they  are  apt  to  redline  them.  Unfamiliarity  with  methods  can  lead 
cost  reviewers  to  question  the  line  item.  Ultimately  a  rudimentary  understanding  of  applicable 
methods  is  important  to  embedding  cognitive  engineering  in  acquisition  practice. 

A  table  of  cognitive  engineering  methods  was  developed  in  phase  I.  The  table  associated 
methods  with  activities  that  were  captured  on  phase  I  flow  charts  that  described  cognitive 
engineering  contributions  to  systems  engineering  products.  These  were,  in  part,  used  to  populate 
the  cognitive  engineering  methods  sections  of  the  deliverables  pages. 
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Methods  were  also  called  out  in  the  “top  10”  list  generated  by  the  project’s  cognitive  engineering 
subcontractor.  They  are  listed  in  table  5.  The  table  5  methods  were  preferentially  selected  over 
those  developed  in  phase  I  because  of  the  empirical  backing  they  had  in  comparison  to  the 
document  research  used  to  generate  the  phase  I  table. 


Table  5:  Methods  from  “top  10”  list 


Artifact  Study 
Behavioral  Task  Analysis 
Cognitive  Task  Analyses 

•  ACTA 

•  ACWA 

•  COGNET 

•  Cognitive  Function  Modeling 

•  Cognitively  Oriented  Task  Analysis 

•  Comprehensive  vs.  Focused  CTA 

•  Concept  Mapping 

•  Contextual  Control  Model 

•  COS1MO 

•  Course  of  Action  Analysis 

•  Critical  Decision  Methods 

•  Decision  Fadder 

•  Decompose,  Network  and  Assess  (DNA) 
Method 

•  Empirical  Framework 

•  Focused  CTA 

•  Goal-Directed  Task  Analysis 

•  GOMS 

•  Grammar  Techniques 

•  Hierarchical  Task  Analysis 

•  Hi-Fo 

•  Interacting  Cognitive  Subsystems  Analysis 

•  KADS 

•  PARI 

•  RPD 

•  Semiotic  Models 


•  Skill-Based  CTA 

•  Sub-Goal  Template 

•  Task  Analysis  for  Error  Identification 

•  Task  Knowledge  Structures 

•  Tasks  Analysis  for  Knowledge  Description 

•  Team  CTA  Techniques 

•  Verbal  Protocol  Analysis 
Contextual  Design  (Work,  Flow,  Cultural, 
Sequence, 

Physical,  And  Artifact  Models) 

•  Interviewing  Techniques 
Contextual  Inquiry 
Cognitive  Work  Analysis 
Ethnography 

Information/Data  Flow  Review 
Instructional  System  Design 
Interviews 

Micro  Saint  Simulation 
Modeling 

Naturalistic  Decision  Making 
Qualitative  Trade  Studies 
Questionnaires 
Scenario  Review 
Simulation 

Situation  Awareness  Oriented  Design 

Stop-Action  Scenarios 

Surveys 

Time-Motion  Studies 
What  iffing? 


There  are  many  repositories  of  cognitive  engineering  methods  in  books,  in  journal  articles  and  on 
the  web.  The  most  accessible  is  MITRE’s  Mental  Models  web  site  (21).  We  planned  to 
introduce  links  to  the  MITRE  site,  because  it  provides  the  seminal  references  for  the  methods. 
One  shortfall  of  the  method  literature  is  that  collections  are  targeted  at  cognitive  engineering 
practitioners  and  are  difficult  for  people  without  an  applied  psychology  to  understand. 
Additionally,  method  descriptions  are  not  neutral;  authors  usually  have  an  agenda  to  promote 
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their  preferred  “school”  or  “method”  over  others.  This  is  confusing  to  systems  engineers  and 
project  managers  who  sincerely  wish  to  include  cognition  in  system  development  and  operation. 
Systems  engineers  and  project  managers  don’t  know  who  to  believe  or  where  to  turn  to  obtain 
the  help  they  need.  Everyone  sounds  like  an  authority. 

Succinct,  20-word-or-less  descriptions  of  the  methods  are  required  for  systems  engineers  and 
project  managers.  It  was  our  plan  to  generate  these,  but  they  have  not  been  completed  as  of  this 
writing. 

Our  team  found  that  it  was  easier  to  ascribe  and  describe  methods  of  other  practitioners  to  the 
deliverables  as  opposed  to  our  own.  The  author  is  a  systems  engineer.  Because  the  systems 
engineering  methods  were  so  ingrained,  taken  for  granted,  it  was  difficult  to  put  a  name  to  what 
was  being  used  to  support  a  deliverable.  On  the  other  hand,  the  cognitive  engineering  colleagues 
were  readily  able  to  identify  the  omission  of  relevant  systems  engineering  methods  from 
associated  deliverables. 

We  also  examined  development  approaches,  philosophies,  that  were  in  keeping  with  the  ways  in 
which  cognitive  engineers  worked.  For  example,  one  attribute  of  cognitive  engineering 
procedures  that  superficially  appears  to  be  at  odds  with  traditional  project  management  is 
continuous  learning  throughout  the  development  cycle.  In  a  continuous  learning  environment 
the  project  must  continually  adjust  and  refine  its  offering  as  new  information  is  garnered.  We 
explored  techniques  that  seem  suited  to  this  paradigm  —  Set-Based  Concurrent  Engineering, 
Rapid  Software  Development  and  Integrated  Modular  [software]  Architecture.  Our  research 
showed  these  techniques  fit  well  in  both  the  systems  engineering  and  cognitive  engineering 
models.  The  techniques  must  be  implemented  by  an  enterprise  through  program  or  project 
management.  It  was  a  weakness  of  the  final  APSE  design  that  our  research  on  these  techniques 
found  no  place  in  the  architecture. 

There  were  also  approaches  that  came  out  of  the  systems  engineering  world  that  are  not 
institutionalized  in  its  practice.  For  example,  Dr.  John  Warfield,  who  for  50  years  has  been 
working  to  better  incorporate  human  needs,  developed  the  Interactive  Management  methodology 
and  Interpretive  Structure  Modeling  software  along  with  his  colleague  Dr.  Alexander  N. 
Christakis  (24).  Christakis  has  documented  the  derivative  structure  design  process  (SDP)  also 
called  the  CogniScope  Approach.  CogniScope  also  employs  software,  CogniSystem. 

The  purpose  of  the  work  of  Warfield  and  Christakis  is  to  lead  stakeholders  through  process  that 
identifies  needs  and  the  root  causes  of  those  needs  to  that  a  logical  action  plan  can  result.  These 
techniques  provide  a  bridge  between  systems  engineering  and  cognitive  engineering  knowledge 
elicitation.  They  are  comprehensive,  but  can  be  time  consuming  and  so  are  underused  to  the 
detriment  of  system  development.  They  are  highly  relevant  to  the  JCIDS  process,  so  we  have 
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incorporated  references  to  the  work  of  Warfield  and  Christakis  in  JCIDS  deliverable 
descriptions. 

Quality  Function  Deployment  (QFD)  is  another  comprehensive,  time  consuming  technique  that 
is  effective  during  concept  development  and  refinement.  It,  too,  is  underutilized.  A  simplified 
version  of  QFD  was  developed  for  the  capstone  design  course  in  space  systems  engineering  at 
University  of  Illinois’  Urbana  campus.  A  cookbook  with  examples  was  prepared  to  walk  users 
through  the  technique.  Templates  and  references  have  been  included  in  JCIDS  documentation. 

An  emphasis  was  placed  on  methods  to  support  JCIDS,  where  gaps  are  identified  and  concepts 
are  developed.  If  human  contributions  to  a  functional  solution  are  incorporated  at  this  early  stage 
of  development,  then  DOTMLPF  analyses  will  be  more  complete  and  solutions  will  incorporate 
people  in  ways  that  enhance  the  mission  without  unnecessarily  driving  up  operations  and 
sustainment  costs.  We  demonstrated  the  strong  correlation  between  DOTMLPF  and  HSI  in  a 
presentation  made  to  the  local  chapter  of  INCOSE.  Aside  from  the  materiel  component, 
DOTMLPF  is  all  about  how  people  are  managed,  organized,  trained,  educated,  selected  and 
accommodated.  These  are  all  topics  treated  by  HSI. 

In  the  course  of  exercising  the  DeSAT  and  SuperSAGAT  software  tools  by  SA  Technlogies,  it 
was  necessary  for  us  to  simulate  a  Goal-Directed  Task  Analysis.  GDTA  was  researched  and  an 
example  was  generated.  The  GDTA  process  resulted  in  situation  awareness  requirements, 
though  the  requirements  were  not  in  a  fonn  usable  by  engineers.  We  used  the  INCOSE 
Requirements  Working  Group’s  dra ft  Requirements  Writing  Guide  (25)  and  translated  GDTA- 
style  requirements  into  actionable  “shall  statements”  to  demonstrate  the  connection  between 
cognitive  engineering  and  systems  engineering.  The  GDTA  example,  with  the  requirements 
translation  tables,  was  an  unplanned  artifact.  We  need  to  identify  how  it  fits  in  the  APSE 
architecture.  Once  we  do,  it  will  be  incorporated  as  an  example. 

3.4.4.2.2.  Software 

For  both  the  freeware  and  professional  confederations  of  analysis  tools,  we  researched  software 
aids  that  could  be  used  as  data  repositories  and  project  management,  systems  engineering,  human 
systems  integration  and  cognitive  engineering  analyses.  On-line  research  was  performed  and 
literature  was  reviewed  for  applicable  software. 

We  discovered  a  number  of  freeware  tools  that  were  appropriate.  Many  were  no  longer 
supported.  We  discovered  tools  that  were  the  subject  of  research  and  development  projects.  We 
were  unable  to  obtain  some  of  those  from  their  developers;  others  did  not  have  a  commercial 
license  agreement  that  would  make  it  possible  to  recommend  them  to  users  who  were  not 
researchers  or  college  students.  Software  vendors,  including  cognitive  engineering  houses, 
touted  aids  on  their  web  sites  that  seemed  relevant  to  the  professional  project.  A  list  of  the 
relevant  software  that  was  uncovered  is  in  Appendix  D. 
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We  had  planned  to  include  a  listing  of  applicable  and  available  software  in  support  of  building 
integrated  design  environments.  We  found  that  such  a  list  provides  little  or  no  value  to  users. 

We  discovered  four  repositories  of  HSI/cognitive  engineering  software  tools  in  web,  document 
and  database  form  and  consulted  the  extensive  web-based  INCOSE  System  Architecture  Tools 
Survey.  The  following  two  on-line  repositories  no-longer  exist:  1)  Navy  Human  Perfonnance 
Center;  2)  National  Defence  and  the  Canadian  Forces.  DTIC  maintains  the  Directory  of  Design 
Support  Methods  (26)  which  holds  human  engineering  tools.  AFRL  conducted  a  survey  of 
NAVSEA  HSI  software  (27)  that  describes  52  software  aids. 

At  the  June  2007  HSIWG  meeting,  the  author  posed  the  question,  “Have  any  of  these 
repositories  helped  to  advance  the  practice.”  After  stunned  silence,  there  was  an  admission  that 
they  don’t  appear  to  have  helped.  The  repositories,  like  the  guidebooks  and  manuals  are  too 
unwieldy  to  use  and  too  voluminous  to  read.  Tool  descriptions  don’t  seem  to  help  users  to  target 
value  applications. 

This  finding  led  us  to  change  our  approach  to  the  software.  We  selected  tools  with  functionality 
that  builds  bridges  between  cognitive  engineering  and  systems  engineering  practice  and 
exercised  them  in  examples,  documented  the  steps  it  took  to  build  a  model  or  a  use  the  software, 
and  provided  our  experience  in  working  with  the  product  for  users. 

The  time  spent  on  each  step  was  recorded.  Software  prices  were  billed  using  the  ranges  shown 
in  figure  17.  Recording  the  time  used  to  set  up  the  exercise  the  software  along  with  its  retail 
price  enables  project  managers  to  budget  for  its  use. 


Freeware/share  ($0) 

<$100 
$100 -$500 
$500 -$1,000 
$1,000  -$5,000 
$5,000 -$10,000 
>$10,000 

Figure  17:  Software  price  ranges 

Software  demonstrated  in  this  narrative  fashion,  including  examples  of  input  and  output  work 
products  provides  a  perspective  user  with  the  infonnation  required  to  determine  whether  it  is 
applicable  to  the  job  they  have  at  hand.  The  stories  illustrate  how  the  tools  provide  infonnation 
that  can  be  used  by  all  of  the  APSE  target  users. 

Demonstrating  software  is  more  expensive  and  time  consuming  than  creating  a  listing  or 
repository,  so  fewer  examples  were  included  in  APSE  than  were  desired.  We  exercised  the 
software  listed  in  table  6. 
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Three  of  the  tools  do  not  appear  in  the  APSE  product.  CMap  Tools  is  a  useful  means  of 
capturing  data  that  could  serve  all  project  stakeholders.  It  was  used  to  construct  the  prototype 
main  pages,  shown  in  the  Interface  section  above.  It  was  not  included  among  our  software 
demonstrations. 


Table  6:  Software  exercised 


Software 

Value  to  APSE  users 

Notes 

CMap  Tools 

Organization  of  project  infonnation;  could  be 
part  of  data  repository 

Not  included  in  APSE 
demonstrations.  Used  for 
prototype  main  pages. 

Micro  Saint  Sharp 

The  2-D  and  3-D  modeling  and  simulation 
capabilities  are  representations  that  can  be 
appreciated  by  systems  engineers,  project 
managers,  and  designers. 

Task  Architect 

Task  bookkeeping  software  useful  for 
capturing  results  of  behavior  and  cognitive 
task  analyses.  Analogous  to  the  systems 
engineering  tools  Doors,  Core  or  Slate. 

TestLog 

Demonstrated  human  aspects  of  product 
testing  that  need  to  be  documented  as  part  of 
test  planning,  overseen  during  text  execution 
and  documented  as  part  of  post- test  analysis. 

DeSAT  and 
SuperSAGAT 

Situation  awareness  based  cognitive 
engineering  tool  that  generates  validated 
requirements 

Product  quality  was  so 
poor  that  incorporating  it 
in  APSE  was  considered 
detrimental  to  project 
goals. 

Model  Center 

Integration  of  independent  analysis  software 

Used  for  uncompleted 

AOA  exercise 

DeSAT  and  SuperSAGAT  are  software  aids  that  instantiate  the  situation  awareness  approach  to 
cognitive  engineering.  The  quality  of  the  software’s  interface  and  functionality  was  poor  and 
would  not  have  shown  cognitive  engineering  off  to  good  advantage  in  the  eyes  of  systems 
engineers.  SA  Technologies  will  shortly  release  version  2  of  the  software  which  is  expected  to 
be  an  improvement.  The  exercise  was  invaluable,  though,  because  it  spurred  the 
interdisciplinary  discussion  of  the  differences  between  infonning  design  and  specifying  design. 
The  discussion  led  to  a  paper  jointly  authored  with  Dr.  Robert  Hoffman  that  was  printed  in  the 
September/October,  2008  issue  of  IEEE  Intelligent  Systems  (28). 

The  Model  Center  exercise  was  initiated  in  support  of  an  Analysis  of  Alternatives  (AO A).  The 
design  team  was  in  the  process  of  building  models  for  inclusion  in  Model  Center  at  the  end  of  the 
contract  performance  period.  As  was  stated  earlier,  we  aggressively  and  ambitiously  pushed  for 
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a  solution  to  the  problem  of  embedding  human  considerations  in  the  acquisition  process.  We 
knew  that  an  actual  AOA  can  take  as  many  as  18  months  for  a  large  program.  With  only  a  few 
months  remaining,  undertaking  even  a  simulated  AOA  was  ambitious.  The  software  was 
installed.  Tutorials  were  completed.  By  executing  the  first  seven  steps  of  the  AOA  process, 
products  were  produced  that  show  the  promise  of  the  approach  taken  to  include  human 
considerations. 

For  the  AOA  demonstration,  a  university  intruder  management  was  selected  as  the  example 
system.  We  completed  six  steps  of  the  Anny’s  eight-step  AOA  process,  see  figure  18.  We  planned 
to  analyze  a  low-capability  and  high-capability  alternatives. 


An  Analysis  of  Alternatives  consists  of  eight  steps 

1 .  Determine  Issues 

2.  Determine  Mission  Tasks 

3.  Determine  Measures  of  Effectiveness  (MOEs) 

4.  Determine  Analysis  Method 

5.  Determine  Alternatives 

6.  Select  Models  and  Data 

7.  Develop  Database 

8.  Perform  the  Analysis 


Figure  18:  Eight-step  AOA  process 

Cognitive  engineers  reviewed  the  AOA  analysis  methods  and  recommended  that  we  incorporate 
the  educational  goals  of  the  classroom  building  which  had  been  previously  excluded  from  the 
work.  This  was  advisable  because  the  study  was  treating  the  construction  of  a  new  university 
building  as  well  as  the  intruder  monitoring  center.  Steps  one  through  four  of  the  analysis  of 
alternatives  were  repeated  to  capture  pedagogical  features  of  the  chemistry  labs. 

The  inclusion  of  human  systems  integration  and  cognitive  design  attributes  illustrated  how  an 
analysis  of  alternatives  could  be  approached  and  how  precursor  data  must  be  prepared  to  enable 
successful  execution.  The  analysis  of  a  university  intruder  response  system  provided  additional 
benefit  from  the  work.  We  felt  this  study  would  be  publishable  on  its  own  merits  and  explored 
publication  avenues. 

We  conducted  an  investigation  of  the  Brahms  software.  Brahms  supports  behavior  task  network 
modeling.  It  differentiates  itself  from  products  based  on  the  Micro  Saint  simulation  engine  but 
being  more  agent -based  which  allows  it  to  better  capture  the  intent  of  the  people  being  simulated. 
However,  it  has  the  same  time-basis  for  assessment  as  do  the  Alion  products,  which  does  not 
help  to  capture  the  desired  improvements  in  performance  that  the  Air  Force  seeks  from  cognitive 
engineering  application.  Brahms,  developed  out  of  NASA  Ames,  is  free  for  research  and  to  the 
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government,  but  there  is  not  a  licensing  approach  for  contractor  use.  This  makes  it  difficult  to 
recommend  Brahms  for  professional  use.  The  Brahms  web  site  (29)  describes  a  product  to 
incorporate  visualization  functionality,  but  this  is  not  part  of  the  available  release  and  must  be 
explored  further. 

We  explored  Trident  Systems  Interchange  SE  with  great  interest.  We  felt  that  if  we  could 
Interchange  SE  as  the  backbone  this  would  help  to  pave  the  way  for  a  demonstration  in  the 
NAVSEA  Navy  Capability  Development  Process.  While  the  Navy  has  not  maintained  use  of  the 
HCDE  in  the  Navy  Capability  Development  Process,  it  has  retained  Interchange  SE.  Interchange 
SE  makes  available  a  central  data  store  with  a  common  interface  that  allows  multiple  commercial 
and  custom  software  tools  to  be  seamlessly  integrated.  When  we  first  explored  incorporation  of 
Interchange  SE  in  the  APSE  presentation  and  discovered  a  single  license  cost  $72,000.  This 
would  have  subsumed  our  entire  software  budget.  We  expressed  this  concern  to  Trident 
Systems. 

Nearly  a  year  later,  we  visited  Trident  Systems  to  discuss  licensing  and  availability  of 
Interchange  SE.  During  our  visit  to  Trident,  we  discovered  that  a  new  Interchange  SE  release 
was  imminent.  This  release,  version  3,  will  be  free  and  downloadable.  The  Interchange  SE 
marketing  plan  now  focuses  more  on  consulting,  training  and  set-up  services.  Interchange  SE 
3.0  is  not  in  a  fonn  which  “an  individual  user  could  be  turned  loose  to  use  it.”  Since  we  did  not 
have  the  budget  to  engage  Trident,  we  opted  instead  to  purchase  Model  Center  by  Phoenix 
Integration. 

3.5.  Market  Development,  Awareness  and  Education 

This  project  exhibited  many  attributes  of  the  design  of  a  socio-technical  system.  Embedding 
cognitive  systems  into  systems  engineering  has  both  a  technical  side,  i.e.  the  model  process  and 
toolset  required  by  the  SBIR  topic,  and  a  human  side  —  people’s  awareness  of  the  emerging 
discipline,  their  familiarity  with  the  value  it  engenders,  their  prejudices  about  what  belongs 
inside  the  envelop  of  a  system  under  consideration,  as  examples.  It  was  interesting  to  note 
people’s  reluctance  to  treat  both  aspects  of  the  socio-technical  system  as  acquisition. 

While  the  color-of-money  considerations  required  that  a  product  result  from  this  effort,  the 
product  had  no  value  to  acquisition  practitioners  without  an  appreciation  of  cost  and  performance 
issues  that  could  not  be  adequately  treated  if  HSI  and  cognitive  engineering  were  not  included  as 
an  integral  part  of  the  acquisition  process.  If  an  a  priori  key  perfonnance  parameter  had  been 
designated  for  this  SBIR  effort,  it  might  have  best  been  stated  as  the  number  of  minds  changed  - 
the  number  of  people  persuaded  of  the  value  of  human  systems  engineering  —  during  the  period 
of  performance.  Of  course,  this  is  a  difficult  parameter  to  measure;  the  authors  can  offer  no 
objective  measure  of  success  against  this  metric.  In  the  following  subsections,  we  describe  the 
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efforts  we  made  to  increase  the  number  of  ticks  on  the  positive  side  of  the  value  balance.  The 
right  side  of  figure  8  shows  the  subsection  titles. 

Presentations  in  2006,  some  predating  conclusion  of  the  phase  II  contract,  what  may  be  called 
“bandwagon”  talks.  People  wanted  to  know  about  the  work  to  introduce  cognition  into 
acquisition;  our  goal  was  to  get  them  involved  as  insiders  as  the  model  process  was  being 
developed.  It  was  better  to  have  them  contributing  to  our  work  as  it  evolved  rather  than 
criticizing  the  work  after  it  was  completed. 

3.5.1.  Meetings 

Since  the  July  2004  Air  Force  Scientific  Advisory  Board  report,  Human  Systems  Integration  in 
Air  Force  Weapon  Systems  Development  and  Acquisition  (30),  individuals  interested  in 
acquisition,  particularly  those  whose  practices  are  related  to  HSI,  have  been  heatedly  discussing 
the  means  the  process  and  means  for  inserting  HSI  into  acquisition.  Our  team  participated  in 
formal  and  informal  meetings  in  which  solutions  were  debated.  We  argued  that  the  human 
aspects  were  as  important,  if  not  more  important,  than  any  technical  solutions  put  forward. 

3.5.1. 1.  Intelligent  Enterprises  Working  Group 

In  2006,  the  author  was  asked  to  present  to  the  INCOSE  Intelligent  Enterprises  Working  Group 
(IEWG).  Tenets  of  IEWG  are  of  interested  to  this  work  because  they  look  at  systems  as 
aggregation  of  people  using  peripherals.  This  perspective  is  in  keeping  with  the  pre-industrial 
revolution  model  discussed  early.  It  is  also  attractive  because  it  establishes  a  continuum  of  sorts 
from  systems  that  are  only  people  to  systems  that  are  principally  machines  run  by  software.  So 
the  same  acquisition  process  could,  theoretically  be  applied  to  a  special  operations  force  as  well 
as  to  an  aircraft  or  naval  vessel.  The  author  reviewed  the  challenges  of  human  centering  to 
provide  improved  performance  and  cost,  the  technical  challenge  of  being  able  to  answer  the  mail 
that  CHI  Systems’  Dr.  Wayne  Zachary  felt  was  of  the  greatest  concern  when  speaking  at  the 
MIT  Humans  and  Technology  Symposium  (31)  which  is  described  in  more  detail  in  section  3.5.2 
The  various  initiatives  to  address  the  challenges  both  in  the  government  and  commercial  sectors 
were  describe,  such  as  the  Pew  and  Mavor  (32)  study  for  the  National  Resource  Council  which 
eventually  turned  out  such  a  confusing  report,  and  Microsoft’s  hiring  of  anthropologists  to  inject 
the  study  of  how  people  use  tools  into  their  product  development  process.  The  cultural  change 
aspects  of  this  project  were  described  -  inclusion  of  all  acquisition  stakeholders  (a  community 
solution),  making  the  change  mechanism  available,  attractive  and  relevant,  and  making  changes 
understandable,  transparent  to  people  affected  by  them. 

Concerns  about  the  seven  +1-2  HSI  domains  was  expressed  even  at  this  early  stage.  HSI  is 
broken  into  nine  domains  -  Manpower,  Personnel,  Training,  Human  Factors,  Environment, 
Health,  Safety,  Survivability,  Habitability.  Some  organizations  do  not  include  habitability. 
Others  bundle  occupational  health  with  environment.  These  were  arrived  at  because  this  is  the 
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way  in  which  anned  service  branches  are  organized.  What  about  other  human  concerns  which 
need  to  be  considered?  For  example,  social  regard  for  design,  are  troops  targeted  more  hostile 
when  their  humvee  has  a  military  design  versus  more  of  a  civilian  look?  How  does  culture  play 
into  design?  Why  aren’t  physiological  and  medical  considerations  included?  How  should  they 
be?  The  latter  omission  could  turn  out  to  be  politically  significant  particularly  for  the  Air  Force 
HSI  Office  as  the  Human  Effectiveness  Directorate  merges  with  the  3 1 1th  Human  Systems  Wing 
to  become  the  71 1th  Human  Perfonnance  Wing. 

Finally,  the  question  of  whether  all  aspects  of  the  human  mind  were  being  considered  was  raised. 
It  was  pointed  out  that  cognition  is  one  of  three  aspects  that  affect  performance.  The  other  two 
are  affect,  or  emotion,  and  the  third  is  conation,  which  can  be  loosely  equated  with  motivation  or 
desire.  Members  of  the  IEWG  resonated  with  this  because  these  are  important  considerations  if 
systems  are  regarded  as  collections  of  humans  using  tools.  Indeed,  military  leaders  recognize  the 
importance  of  affect  and  conation,  but  these  two  aspects  of  the  mind  do  not  seem  to  be 
considered  important  in  the  design  of  military  systems. 

3.5.1.2.  HFES  2006 

Just  as  the  prime  contractor  networked  with  fellow  systems  engineers,  Klein  Associates,  the 
cognitive  engineering  subcontractor  to  this  effort,  performed  similar  awareness-raising  at  HFES. 
The  networking  was  more  one-on-one  than  the  presentation  made  to  the  IEWG,  but  was 
similarly  designed  to  familiarize  colleagues  with  ongoing  activities  and  issues. 

3.5.1.3.  Klein  Associates  Brownbag  Sessions 

Starting  in  March,  2007,  systems  engineers  were  invited  to  participate  in  lunch  time  discussions 
hosted  by  the  cognitive  systems  engineering  group  of  Klein  Associates.  Discussions  were  about 
improvement  of  cognitive  [systems]  engineering  practice  and  how  to  increase  the  scope  and 
scale  of  the  practice  by  expanded  contributions  to  acquisition.  Topics  included: 

•  APSE.  A  presentation  was  made  and  fonnative  feedback  was  obtained. 

•  A  cognitive  engineering  training  seminar  consisting  of  six  half-day  modules  that  could  be 
selectively  presented  for  a  target  audience.  A  version  was  developed  and  presented  at  the 
2008  HFES  Annual  Meeting.  Additional  funding  was  sought  in  order  to  turn  it  into  a 
regular  event  similar  to  the  human  factors  training  provided  by  University  of  Michigan. 

•  Certification  of  the  cognitive  engineering  training  seminar  by  INCOSE  so  it  could  be 
presented  regularly  to  systems  engineers. 

•  Certification  of  cognitive  engineering  practitioners.  HFES  past  president  Dr.  Marv 
Dainoff  was  a  member  of  the  human  factors  certification  board  and  was  querying  the 
group  about  skills  and  abilities  that  would  be  distinguishing. 

•  What  would  go  on  the  short  list  of  essential  cognitive  engineering  references? 
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•  Blogs  and  list  serves  as  communication  and  education  mechanisms. 

•  Establishment  of  a  cognitive  systems  engineering  speakers  bureau. 

•  The  Cognitive  Systems  Engineering  Landscape.  Four  participants  in  the  brownbag 
sessions,  Dr.  Cindy  Dominguez,  Dr.  Gary  Klein,  Dr.  Gavan  Lintern,  and  Dr.  Laura 
Militello  have  been  working  for  several  years  on  an  overview  paper  of  cognitive  systems 
engineering.  They  were  encouraged  to  complete  it  for  a  special  issue  of  Cognitive 
Engineering  and  Decision  Making  being  edited  by  Dr.  Richard  Pew  and  Dr.  Emily  Roth. 
The  deadline  passed  before  edits  were  completed.  Dr.  Militello  was  offered  payment 
under  this  contract  if  that  would  help  to  complete  the  article. 

The  paper  was  conceived  to  be  a  general  explanation  of  cognitive  systems  engineering 
for  lay  people.  In  the  past,  fine  points  included  about  a  particular  school  or  philosophy  of 
cognitive  engineering  rendered  the  article  and  articles  like  it  opaque  to  systems  engineers 
and  project  managers.  At  HSIWG  gatherings,  systems  engineers  still  didn’t  know  what 
cognitive  engineering  was;  they  were  not  prepared  for  a  discussion  of  its  finer  points. 

In  December  2008,  after  some  encouragement  including  the  offer  of  payment,  the  edits 
had  been  made  and  the  article  was  being  finalized  by  its  authors.  A  slimmed  down 
version  has  been  solicited  for  the  upcoming  INCOSE  Insight  publication  on  cognition. 

The  participation  of  the  systems  engineers  present,  this  report’s  author  and  Mr.  Michael  Mueller 
of  the  Air  Force  Center  for  Systems  Engineering,  helped  to  turn  this  group  into  one  that  was 
more  outward  facing.  Outward  facing  means  the  group  is  addressing  its  customers  as  opposed 
to  people  who  practice  within  the  same  specialty.  We  captured  and  distributed  minutes,  helped 
to  shape  seminar  content,  and  encouraged  the  completion  of  important  publications. 

3.5.I.4.  HSIS  2007 

In  March  2007,  the  American  Society  of  Naval  Engineers  hosted  HSIS,  HSI  Symposium.  HSIS 
“provide[s]  a  forum  for  HSI  experts  from  military,  industry,  and  academia  to  exchange 
information  on  emerging  military  systems,  promising  research,  and  the  benefits  of  effective  HSI 
implementation.  This  conference  seeks  to  share  lessons  learned  from  military,  industry,  and 
academia  to  support  future  improvements  in  design  processes  and  systems.”  (33) 

This  was  primarily  an  intake  session.  We  engaged  display  vendors  to  understand  their  practice 
and  their  offerings  to  try  to  understand  how  they  contributed  to  the  developing  model  process 
and  how  APSE  fit  into  the  mix  of  products  and  services.  The  two  most  important  insights  were 
the  following:  1)  the  importance  of  the  lexical  challenges  of  cross-disciplinary  practice  and  2) 
the  inward  looking  posture  of  the  HSI  community.  Just  as  with  the  brownbag  sessions  and 
INCOSE,  practitioners  find  more  comfort  in  talking  with  their  peers  than  with  people  who  can 
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receive  value  from  their  products  and  services.  The  latter  insight  strongly  influenced  our 
subsequent  efforts. 


3.5.I.5.  Beyond  Phase  II  Conference 

In  August,  2007  the  National  Defense  Industry  Association  sponsored  Beyond  SBIR  Phase  II 
Conference  &  Exhibition  2007:  Bringing  Technological  Edge  to  the  Warfighter.  The  conference 
allowed  SBIR  participants  to  arrange  15-minute  sessions  with  prime  contractors,  government 
acquisition  managers,  the  investment  community,  and  manufacturing.  Ten  appoints  were  set  up. 
Table  7  lists  them. 


Table  7:  Beyond  phase  II  conference  meeting  summaries 


Contact 

Outcome 

Mr.  Martin  E. Trujillo 

Liason  to  SPA  WAR  PMW-180 

PEO  C4I  &  I/O 

Sent  him  APSE  flyer.  Trujillo  said  he  would  send  email 
around  his  organization.  “Can  only  lead  a  horse  to 
water...” 

Mr.  Jack  Griffin 

NAVSEA  SBIR  Program 

Coordinator  Undersea  Warfare  Center 
Division 

Interested  in  setting  up  a  CRADA  to  have  NAVSEA 
systems  engineers  test  use  of  APSE.  Main  purpose  for 
us  was  to  obtain  a  testimonial  about  the  product  for 
marketing  use.  CRADA  abandoned  on  Navy  side. 

Andre  Valente,  PhD 

Chief  Operating  Officer 

Tactical  Language  and  Culture 

Dr.  Valente  was  interested  in  exploring  cross-cultural 
training  product  in  context  of  HSI.  Explained  contract 
and  introduced  him  to  HSIWG.  No  follow-up. 

Raj  K.  Aggarwal,  PhD 

Vice  President 

Global  Technology 

Engineering  and  Technology 

Rockwell  Collins 

He  didn’t  really  see  how  it  fit  in  with  what  they  did. 

Asked  if  we’d  met  with  systems  engineers  and  cognitive 
engineers  to  determine  if  these  were  the  right  questions 
we  need  to  be  answering.  Mentioned  the  phase  I  Glen 
Helen  meeting.  Suggested  working  with  their 
organizations  Linda  Simmons. 

William  A.  (Bill)  Freiberg 

Capture  Manager 

Advanced  Combat  Aircraft  Systems 
Phantom  Works 

The  Boeing  Company 

Said  this  was  fascinating  stuff. 

At  first  he  said  they  don’t  use  systems  engineers,  but 
then  he  said  that  as  part  of  the  capability  definition  and 
design  they  do  Design  of  Experiments  which  are 
supported  by  systems  engineers. 

Thought  APSE  fit  into  the  “Lean”  buzzword.  He 
triggered  off  the  high  value  activities  to  make  the 
connection  between  lean. 

He  saw  how  this  could  support  them  in  what  they  did. 

Said  someone  would  follow  up  with  me,  perhaps  Steve 
D’Urso  (UIUC  alumni  president).  No  follow  up  took 
place. 
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Table  7:  Beyond  phase  II  conference  meeting  summaries  (cont’d) 


Contact 

Outcome 

Stanley  U.  Levy 

and 

Larry  Butler 

HMS/Maintainability/T  estability 
Product  Integrity  Engineer 

Space  and  Airborne  Systems 
Engineering 

Raytheon 

Larry  is  a  systems  engineer.  Couldn’t  understand  where 
the  requirements  came  for  the  kind  of  things  APSE  was 
talking  about.  He  said  they  got  MIL  STD  1472,  Human 
Factors,  which  he  said  he  could  quote.  With  the  IPT 
structure  chart,  he  got  the  connection  to  MANPRINT. 

I  told  him  about  NAVPRINT,  AIRPRINT  and 
JOINTPRINT.  He  said  they  didn’t  do  this  very  well. 

They  have  a  guy,  Bob  Schwam  who  is  a  human  factors 
guy.  He  thought  Bob  would  be  willing  to  review  this  as 
part  of  an  internal  study.  Need  to  better  understand  the 
PRINTS  if  these  requirements  are  coming  down  the 
pike.  I  showed  them  Dennis  Folds’  charts  and  they  were 
interested. 

Sent  one-page  APSE  flyer  to  Larry  and  Stan  along  with 
Dennis’  charts  on  HSI  and  the  link  to  our  HSI  web  page. 

Andrew  Bodkin 

Principal 

Bodkin  Design  &  Engineering 

No  use  for  APSE. 

Ron  Szymanski 

US  Army  CERDEC 

Suggested  we  contact  DAU  and  show  off  APSE  perhaps 
as  a  complement  to  the  courses  they’re  providing. 

(Note:  Jack  Griffin  made  the  same  suggestion.)  I 
showed  him  Dennis  Folds’  charts  from  the  HSI  seminar. 
Sent  him  a  copy  of  them. 

Daniel  (Dan’l)  S.  Thomas 

Program  Manager 

Detection  Systems 

General  Dynamics 

No  use  for  APSE. 

Michael  Zarnmit 

Missile  Defense  Agency 

SBIR/STTR  Program  Manager 

Mike  asked  me  to  check  the  MDA  6.3  SBIR 
announcement  for  Human  Effectiveness  topics.  He 
thought  sure  there  were  some.  He  suggested  that  I  get 
the  contact  names  from  those  offerings,  send  them  to 
him  with  the  one  page  flyer  about  APSE  and  he  would 
get  it  into  those  people’s  hands.  Mike  sent  me 
references,  but  no  contact  names  were  listed. 

Each  of  the  interviews  raised  the  awareness  of  the  HSI  and  cognitive  engineering  in  the  minds  of 
the  contact  individuals.  They  were  a  representation  of  the  audience  that  must  be  reached.  Some 
people  heard  about  HSI  for  the  first  time.  Others  saw  no  relevance.  Some,  like  Raytheon’s 
Larry  Butler,  were  aware  of  human  factors  requirements,  but  were  unfamiliar  with  the  broader 
practice  of  HSI  and  had  not  been  introduced  to  cognitive  engineering  at  all. 
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A  panel  made  up  of  the  SBIR  leaders  from  the  Air  Force,  Anny  and  Navy  opened  the  conference 
sessions.  At  the  conclusion,  they  took  questions.  This  report’s  author  asked  the  first  question 
which  was,  “What  if  the  solution  to  a  capability  need  stated  in  a  SBIR  is  not  technical,  but, 
instead,  is  related  to  changes  in  human  performance  much  in  the  manner  that  a  DOTMLPF 
solution  could  be  an  alternative  to  a  materiel  acquisition?”  Each  responder  was  able  to  describe 
the  importance  of  humans  and  HSI  in  their  organization,  but  did  not  directly  relate  the 
importance  to  manifestations  in  the  SBIR  program.  Two  more  of  the  remaining  five  questions 
taken  by  the  panel  related  to  the  incorporation  of  humans  in  capability  solutions.  This  created  a 
buzz  among  conference  attendees.  HSI  and  cognitive  engineering  became  topics  of  break  and 
lunchtime  discussions  for  the  following  two  days.  Unfortunately,  the  exhibiting  vendors  and 
people  like  those  listed  in  table  7  were  unaware  of  this  because  they  were  not  able  to  attend 
presentation  sessions  and  took  their  breaks  and  lunches  away  from  the  other  attendees.  This  is 
why  the  face-to-face  interviews  were  important  educational  opportunities  from  the  perspective  of 
SBIR  AF05 -071. 

As  stated  above,  the  social  engineering  of  the  cultural  change  required  for  a  solution  to  the 
problem  posed  by  this  topic  was  primarily  accomplished  by  changing  one  mind  at  a  time.  The 
question  to  the  panel  and  the  interviews  were  opportunities  that  were  sought  or  created  as  part  of 
our  approach. 

3.5.I.6.  CHI/IHMC 

October  2007,  CHI  Systems  and  the  Institute  for  Human  Machine  Cognition  (IHMC)  hosted  a 
two-day  workshop  titled  Merging  Cognitive  Systems  Engineering  into  Systems  Engineering: 
Implications  for  Large-Scale  Information  Systems  Procurement  (34).  “The  Workshop  will 
promote  discussions  that  consider  real  world  constraints  and  challenges  in  the  procurement  and 
design  of  human  centered  technologies.  Participants  will  discuss  lessons  learned  and  their  best 
ideas  about  how  to  fix  “the  system.”  The  Workshop  will  take  steps  toward  creating  a  roadmap 
for  human-centered  procurement  that  goes  significantly  beyond  current  guidance  (e.g.,  DoD 
Instruction  5000.2R).”  (34) 

Workshop  participants  appeared  to  align  themselves  into  researchers  and  people  with  experience 
in  product  development.  Researchers  spoke  eloquently  about  theories  of  complexity  and 
resilience  and  affordances.  These  words  confused  the  development  community  who  didn’t 
understand  how  what  the  researchers  were  propounding  fit  within  the  world  of  development. 
When  the  simple  definition  of  HSI  (see  below)  was  challenged  by  researchers,  the  author  pointed 
out  that  “No  one  outside  this  room  cared  at  all  about  the  theories  which  were  being  put  forward 
by  researchers.”  The  development  people  affirmed  this  statement.  While  project  managers  and 
the  engineering  community  see  no  value  or  have  no  appreciation  of  human  integration,  the 
theories  had  no  place  to  in  practice.  It  became  apparent  that  research  community  representatives 
were  advocating  the  need  for  more  research  dollars  in  order  to  “fix  the  system.” 


48 


The  PowerPoint  (figure  30,  Appendix  C)  paper  prototype  of  APSE  was  presented  to  the  group. 
There  was  not  universal  enthusiasm,  but  several  people  wanted  the  product  immediately  and 
were  even  interested  in  acquiring  access  to  the  early  Backpack  prototype.  Some  of  the 
requestors  appreciated  the  content.  Others  embraced  it  as  a  planning  tool. 

3.5.2.  Papers  and  Presentations 

3.5.2. 1.  MIT  Humans  &  Technology  Symposium 

In  January,  2006,  MIT’s  Humans  and  Automation  Laboratory  sponsored  the  Humans  & 
Technology  Symposium.  Human  factors,  cognitive  engineering  practitioners  and  students 
participated  in  this  three-day  study  that  took  place  between  the  phase  I  and  phase  II  activities  of 
the  SBIR  topic.  The  presentation  reviewed  the  operational  challenges,  recent  history  of 
integrated,  concurrent  software  environments  and  spoke  about  the  integration  of  disciplines. 

3. 5. 2.2.  The  Ninth  World  Conference  on  Integrated  Design  and  Process  Technology 

At  the  invitation  of  Jack  Ring,  a  paper  was  prepared  and  presentation  was  made  at  the  IPDT 
Conference.  The  paper’s  abstract  is  below. 

“Military  and  commercial  entities  throughout  the  world  are  recognizing  that  systems  which  take 
advantage  of,  and  do  not  impede,  human  cognitive  capabilities  deliver  improved  performance. 
The  United  States  Air  Force,  Army  and  Navy  have  funded  studies  to  investigate  and  develop 
integrated,  human-centered  design  support  methodologies  and  tool  sets.  This  presentation 
provides  an  overview  of  those  efforts  and  speaks  specifically  to  work  funded  by  Air  Force 
Research  Laboratory’s  SBIR  program.  Designing  systems  that  include  human  participants 
within  the  system  boundary  is  not  a  new  topic;  it  has  been  under  study  by  those  interested  in 
systems’  realization  for  over  three  decades.  However,  customary  practice  defines  humans’ 
capabilities  and  concerns,  their  development,  and  their  rotation/promotion  cycles  to  be  outside 
the  boundary.  This  hinders  the  holistic  consideration  of  the  continuum  from  wholly  materiel 
systems  to  systems  comprised  entirely  of  human  components.  Procedural  and  cultural  barriers  to 
a  cost-effective,  implementable  process  have  been  identified  and  will  be  described.  Criteria  for 
developing  a  tool  set  to  support  a  process  that  meets  institutional  needs  without  constraining 
competition  will  be  outlined.  A  transition  scenario  for  satisfying  the  DoD’s  near  tenn  vision  of 
practice  will  be  elucidated.  Finally,  the  need  for  a  long-term  evolutionary  process  vision  will  be 
discussed.”  (36) 

The  presentation  was  a  more  elaborate  version  of  the  informal  briefing  that  was  given  to  IEWG. 
Air  Force,  Anny  and  Navy  HSI  efforts  were  described.  This  SBIR  effort  and  participating 
companies  were  introduced.  The  Pew  and  Mavor  NRC  study  was  shown.  Web  sites  with 
reference  materials  for  interested  individuals  were  shared.  The  two  thrusts  of  the  activity, 
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technology  for  “answering  the  mail”  and  cultural  change  for  creating  value,  were  highlighted. 
Barriers  to  success  and  paths  to  resolution  were  described.  Principal  among  these  paths  was  an 
early  description  of  APSE.  This  was  to  make  people  aware  that  the  product  was  coming.  The 
HSIWG  was  also  introduced  along  with  contact  information  for  people  who  wanted  to 
participation  in  the  group. 

3.5.2.3.  IEEE  Computer 

Discussions  at  the  Klein  Associates  brownbag  sessions  modified  the  perspectives  of  cognitive 
engineering  pioneer  Dr.  Gary  Klein.  The  thesis  or  the  article  was  “Cognitive  systems 
engineering  is  a  value-added  technology  offering  many  benefits  that  outweigh  its  costs.”  (35). 
The  paper  was  a  breakthrough  in  the  sense  that  it  was  reaching  beyond  the  cognitive  engineering 
community  into  arenas  where  that  value  could  be  realized  by  customers.  Cognitive  systems 
engineering  was  defined  as  “the  effort  to  support  the  cognitive  requirements  of  work.”  This 
definition  was  first  crafted,  to  our  knowledge  by  the  paper’s  co-author  and  participant  in  this 
effort,  Sterling  Wiggins.  It  is  the  simplest  and  most  easily  comprehended  of  all  those  that  were 
uncovered  during  this  study. 

Another  idea  that  came  out  of  the  brownbag  discussions  was  the  value  cognitive  engineering 
could  bring  to  project  management.  Cognitive  engineers  have  expertise  in  team  design  and 
decision  making  that  could  help  managers  to  tailor  program  structures.  Additionally,  their 
knowledge  elicitation  skills  demonstrably  helped  to  improve  the  efficiency  of  meetings  on  the 
programs  like  DDX. 

3.5.2.4.  IEEE  Intelligent  Systems 

Discussions  about  the  influential  differences  between  “informing”  design,  the  approach  taken  by 
cognitive  engineering  and  “specifying  design”  that  arose  from  our  exercise  of  Goal  Directed 
Task  Analysis  resulted  in  the  publication  of  a  paper  jointly  authored  with  Dr.  Robert  Hoffman 
(28).  The  importance  of  replicas  or  “shall  statements”  for  directing  software  development  was 
emphasized.  Dr.  Hoffman  described  how  this  can  be  achieved  as  part  of  cognitive  task  analysis. 

3.5.2.5.  2nd  International  Conference  on  Applied  Human  Factors  and  Ergonomics 

At  Dr.  Marv  Dainhoff  s  invitation,  a  paper  was  prepared  and  presentation  was  given  for 
AHFEI.  The  session  theme  was  HSI.  The  focus  of  the  paper  was  on  the  HSIWG  and  the 
progress  it  had  made.  The  abstract  follows. 

“Since  February,  2006,  the  author  has  served  as  co-leader  of  the  International  Council  on 
Systems  Engineering’s  Human  Systems  Integration  Working  Group  (INCOSE’s  HSIWG). 
The  HSIWG ’s  purposes  are  to  facilitate  embedding  human  systems  integration  within 
systems  engineering  and  to  promote  the  benefit  of  placing  the  proper  focus  on  the  role  of 
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people  in  the  development  and  operation  of  systems.  This  paper  explains  that  purpose, 
describes  the  group’s  progress  and  accomplishments,  examines  the  barriers  to  success,  and 
explores  future  opportunities  for  the  working  group,  for  societies  of  professionals  and  for 
individual  practitioners.”  (36) 

The  briefing  focused  on  the  need  to  look  outward,  to  connect  with  customers  up  the  demand 
chain  and  be  aware  of  what  was  being  provided  through  the  supply  chain.  The  need  for  all 
stakeholders  in  the  APSE  chain  (from  program  managers  to  cognitive  engineers)  to  illustrate 
the  value  of  the  services  they  provide  was  emphasized.  Fifty  copies  of  INCOSE  Insight 
special  HSI  edition  (5)  were  distributed  to  session  and  conference  attendees.  A  list  of  the 
APSE  deliverables  was  also  provided  as  a  handout. 

3.5.3.  HSIWG 

This  projects  topic  statement  implies  that  systems  engineering  is  the  accepted  state  into  which 
cognitive  engineering  is  to  be  inducted.  In  order  to  achieve  the  desired  penetration  into  systems 
engineering  process,  a  working  group  was  initiated  within  the  technical  structure  of  the  INCOSE. 
The  group  was  chartered  under  a  systems  engineering  enabler  function  which  employs  a  very 
narrow  definition  of  specialty  engineering  to  include  manufacturability,  cost,  reliability, 
serviceability  -  all  the  ‘ilities.  Starting  from  two  members  in  January  2006,  the  group  has  grown 
to  over  150  participants.  It  meets  twice  a  year  at  the  International  Workshops  in  January  for  four 
days  and  at  the  INCOSE  International  Symposium  in  June  or  July  for  a  half  day  session. 

The  first  task  of  the  group  was  to  establish  purpose  and  vision  statements.  The  author  moderated 
a  gathering  of  nineteen  people  who  crafted  the  following  statements. 


PURPOSE:  The  INCOSE  Human  Systems  Integration  Working  Group  will  facilitate 
embedding  HSI  within  Systems  Engineering,  promoting  the  benefit  of  placing  the  proper  focus 
on  the  role  of  people  in  the  development  and  operation  of  systems. 

VISION:  HSI  is  embedded  in  SE  practices,  leading  to  the  efficient  delivery  of  effective 
systems. 

The  following  year,  a  succinct  definition  of  HSI  was  crafted.  The  author  put  together  a  white 
paper  that  captured  49  known  definitions  of  HSI.  At  the  workshop  sessions,  a  group  of  35 
whittled  the  definition  down  to  21  words.  From  January  through  April,  listserve  discussions 
were  conducted  to  refine  and  gain  acceptance  for  a  definition.  The  resulting  26-word  definition 
is  below. 

HSI  is  the  interdisciplinary  technical  and  management  processes  for  integrating  human 
considerations  within  and  across  all  system  elements;  an  essential  enabler  to  systems 
engineering  practice. 

This  is  the  definition  that  was  seen  as  too  simplistic  by  researchers  at  the  CHI/IHMC  gathering  in 
October,  2007.  What  they  did  not  appreciate  was  systems  engineers  did  not  universally  agree 
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that  human  considerations  belonged  inside  the  system  design  envelope.  The  INCOSE  definition 
eliminated,  while  not  having  the  ability  to  change  people’s  minds,  became  an  official  statement 
that  essentially  said,  ‘Humans  are  a  part  of  the  system.’ 

Another  important  aspect  of  the  definition  was  that  it  did  not  state  that  systems  engineers  were 
going  to  execute  human  systems  integration  tasks  or  become  the  experts.  This  was  an  important 
consideration  because  the  group  did  not  want  to  alienate  those  who  had  the  skills  required  to 
perfonn  the  work. 

The  question  must  be  asked,  if  the  SBIR  topic  required  researchers  to  address  cognition,  why  the 
emphasis  on  HSI  in  the  systems  engineering  community?  As  commonly  proscribed,  HSI 
includes  the  domains  shown  in  figure  19.  Cognitive  engineering  is  a  subset  of  human  factors 
engineering.  In  practice,  cognitive  engineering  impacts,  manpower,  personnel,  survivability, 
habitability,  team  dynamics,  and  the  sensory  aspects  of  human  factors.  Cognitive  engineering  is 
also  instrument  in  development  training  requirements.  So  its  influence  permeates  HSI  practice 
but  is  not  identical  to  it.  Equating  HSI  with  cognitive  systems  engineering  was  the  mistake  made 
in  the  Pew  and  Mavor  NRC  report  (32).  The  convolving  of  the  two  terms  has  resulted  in 
confusion  and  an  unfortunate  backwash  against  cognitive  engineering. 


Figure  19:  HSI  domains  and  the  relationship  of  cognitive  engineering 
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3.5.3. 1.  Education 


3.5.3. 1.1.  Dr.  Dennis  Folds,  Georgia  Tech  Research  Institute 

At  the  suggestion  of  then  director  of  the  Air  Force  HSI  Office,  Dr.  Richard  Drawbaugh,  Dr. 

Folds  was  invited  to  present  a  seminar  on  FISI  practice  at  INCOSE’s  2007  International 
Symposium.  INCOSE  leadership  paid  Dr.  Folds’  speaking  fee  on  this  first  occasion.  At  that 
time,  HSI  participants  were  advancing  their  own  theories  and  opinions  about  HSI  practice.  We 
wanted  to  hear  from  an  actual  HSI  practitioner  to  understand  how  it  worked.  Dr.  Folds  has 
experience  in  all  the  figure  19  domains  save  Habilitability.  He  spoke  for  a  full  day  and  after 
made  his  presentation  charts  available  for  posting  to  the  web.  They  are  available  from 
http://www.incose.org/practice/techactivities/wg/hsi/  and  provided  source  material  for  much  of 
the  HSI  APSE  content. 

The  following  January,  Dr.  Folds  spoke  for  a  half-day  at  the  2008  International  Workshops.  This 
time,  his  topic  focused  on  HSI  Analysis.  Dr.  Folds’  speakers  fee  was  funded  by  this  contract 
effort  as  INCOSE  declined  to  pick  up  his  fee.  As  with  the  2007  seminar,  Dr.  Folds  provided  his 
slides  for  posting,  but  we  were  unsuccessful  in  getting  INCOSE  volunteers  to  post  the  charts  on 
the  open  HSIWG  web  site.  They  were  made  available  to  INCOSE  members  through  the  working 
group’s  SharePoint  web  site  and  can  be  accessed  via  APSE.  This  presentation  also  provided 
source  material  for  the  HSI  APSE  content. 

3.5.3. 1.2.  Dennis  Carlson,  Pit  Stop  Engineering 

Dr.  Mary  L.  Lozano,  Ph.D,  an  anthropologist  working  for  Northrop  Grumman  Electronic 
Systems,  recommended  Mr.  Carlson  as  speaker  for  HSI.  She  touted  his  unique,  philosophy  of 
design  which  places  “humans  first,  machine  second”  as  the  best  way  to  achieve  mission 
performance  goals.  Mr.  Carlson,  an  award-winning  designer,  takes  an  approach  that  differs  is 
from  all  others  that  were  encountered  during  this  effort.  Motivated  by  his  experience  in 
NASCAR  design,  he  wraps  systems  around  the  people  who  use  them.  Mr.  Carlson  provided  a 
half-day  seminar  to  the  HSIWG. 

A  primary  feature  of  his  presentation  was  his  inclusion  of  testimonials  which  subjectively 
documented  people’s  satisfaction  with  his  work,  but  also  supplied  numbers  regarding  the 
operations  costs  saved.  We  were  advised  by  marketers  we  consulted  that  testimonials  were 
essential  for  advertising  the  value  of  APSE  for  our  target  audiences.  Mr.  Carlson’s  presentation 
dramatically  illustrated  their  power  and  influence. 

3.5.3.2.  Alliances 

3.5.3.2.I.  HFES-INCOSE  Memorandum  of  Understanding 

A  memorandum  of  understanding  (MOU)  was  drafted  in  the  fall  of  2007.  It  was  submitted  to  the 
HFES  executive  director  in  October  of  that  year.  Shortly  after  the  submission,  the  HFES  System 
Development  Technical  Group  (SDTG)  made  modifications  to  its  charter  which  positioned  it  as 
the  leader  HSI  technical  group  within  HFES.  The  charter  revision  provided  an  immediate  point 
of  intersection  between  the  two  groups  and  cross-over  discussions  were  initiated  at  the  2007 
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HFES  annual  meeting.  This  arrangement  created  a  bridge  between  HSI  and  cognitive 
engineering  practitioners  and  systems  engineers  which  will  help  members  of  this  practitioner  sets 
to  understand  what  processes,  methods  and  tools  are  needed  to  have  to  make  an  effective  impact 
on  mission  perfonnance  and  ownership  costs. 

The  MOU  went  through  a  review  process  and  revision.  The  revised  version  was  submitted  to 
INCOSE  leadership  one  year  later.  The  MOU  is  intended  to  establish  a  relationship  between  the 
leadership  of  the  two  organizations  so  joint  strategies  can  be  undertaken. 

3.5.3.2.2.  IEEE  Systems,  Man,  and  Cybernetics  Society 

IEEE  SMC  vice  president  Dr.  Ellen  J.  Bass  was  invited  to  present  to  the  HSIWG  at  the  2008 
international  workshops.  The  goals  of  SMC  are  similar  to  those  of  INCOSE,  though,  like  HFES, 
the  emphasis  is  more  on  how  to  practice  and  how  to  improve  practice  than  in  extending  the 
practices  of  SMC  members.  Dr.  Bass  was  open  to  joint  activities.  HSIWG  leadership  transition 
slowed  the  process.  Dr.  Bass  attended  the  2008  AHEI  meetings  in  July  at  which  a  student  of 
hers  presented.  Informal  discussions  of  linked  activities  were  continued  after  the  session. 

3.5.3.3.  Publications 

3.5.3.3.1.  Appendix  to  Systems  Engineering  Handbook 

Members  of  HSIWG  contributed  to  an  HSI  appendix  to  the  INCOSE  Systems  Engineering 
Handbook  version  3.1  (2).  Not  only  does  this  have  value  for  the  influence  it  has  over  acquisition 
policy,  it  also  brings  the  handbook  into  line  with  the  content  of  IEEE  1220  which  is  laced  with 
references  to  human  systems  engineering  practice. 

3.5.3.3.2.  Integrating  the  Human  in  Every  System 

In  April  2008,  HSIWG  sponsored  a  special  edition  of  INCOSE  Insight  magazine  (5).  The 
magazine  is  distributed  to  the  more  than  5,000  INCOSE  members.  Fifty  copies  were  also 
distributed  at  AHEI  and  100  were  provided  to  individuals  at  the  Air  Force  Center  for  Systems 
Engineering  by  request.  Four  hundred  copies  were  provided  to  the  vendors  at  EITSEC  2008. 
Table  8  lists  the  contents,  authors  and  topics  contained  in  the  issue. 
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Table  8:  Contents  of  the  Insight  special  edition  on  HSI 


Section/Article 

Author(s) 

Emphasis 

“The  Pervasive, 
Indispensable  Human  ” 

Michael  Mueller 

Issue  Overview 

Humans  impact  all  systems 

Extension  of  Specific 

HCI  Methods  and  Tools 
for  Higher-Level  HSI 
Application 

Major  Nick  Hardman 

Lt  Col  John  Colombi 

HSI  Methods  &  Tools 

HSI  Technology 

Building  to  the  HSI 
Demonstration 

Dennis  Folds 

HSI  methodology  and  tools 

HSI  analyses 

HSI-systems  engineering  synergies 

Talking  the  Talk  -  Cross- 
Discipline  Terminology 
Challenges 

Jennifer  Narkevicius 

John  Winters 

Impacts  of  education  and  training 
segregation  on  language  and  practice 
Methods  for  successfully  bridging  the 
multiple  disciplines  (domains,  systems 
engineering,  HSI,  PM,  funding) 

JPRINT  Overview 

Jen  Narkevicius, 

John  Lockett,  and 
Gretchen  Lizza 

HSI  Policy  -  evolution  and  needs 

HSI  organization 

HSI  technology  forecasting  and 
prioritization 

HSI  training  and  education 

HSI  is  not  just  for 
Department  of  Defense  - 
Contrasting  HSI  Practice 
in  Military  and 

Commercial  Sectors 

John  Winters  et  al 

HSI  methodology  and  tools 

HSI  in  organization 

HSI  Policy 

HSI  economics 

Training  and  education 

HSI  in  Commercial  Ship 
Design 

Alexander  C.  Landsburg 

HSI  in  lifecycle 

Economics  of  HSI 

HSI  Tools  and  methodolgy 

HSI  in  organizational  structures 

10  Best  Practices  of  HSI 

Editors 

Summary  pulled  from  the  other  articles 
and  author  recommendations 

Sidebar  of  HSI  Resources 

Editors 

3.5.3.3.3.  Cognition:  Pursuing  the  Next  Level  in  System  Performance 

A  second  special  edition  of  Insight  magazine  is  in  preparation.  This  issue’s  theme  is  cognition. 
It  will  go  to  press  in  April,  2009.  Authors  have  been  approached  and,  with  some  exceptions, 
have  been  confirmed.  As  can  be  seen  from  the  planned  contents  given  in  table  9,  the  emphasis 
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of  the  issue  is  on  successes  -  what  has  worked  in  practice.  The  issue  is  intended  to  show 
examples  of  how  systems  and  cognitive  engineers  have  worked  together  successfully  in  the  past 
and  provide  tips  on  how  it  should  be  done. 


Table  9:  Contents  of  the  Insight  special  edition  on  cognition 


Section/Article 

Author(s) 

Emphasis 

Editor's  intro  —  Deal 

Deal 

Overview  of  issue 

Why  Cognitive  Engineering  is 
Important? 

TBD 

Customer  demand  focus 

The  Cognitive  Systems  Engineering 
Landscape 

Laura  Militello 

Definition  of  domain 

Situation  Awareness 

Laura  Strater 

Success  in  applying  situation 
awareness  methodologies 

Modeling  User 

Clyde  Wetteland 

Overview  of  Alion  successes 
in  behavior  and  cognitive 
modeling 

TBD 

Paul  Picciano 

Successful  use  of  Aptima 
tools 

Submarine  Design 

Sterling  Wiggins 

Successes  in  team  design  of 
submarine  control  rooms 

Pistop  Engineering 

Dennis  Carlson 

Successes  in  human  first, 
machine  second 

Architectures  and  Cognition  — 

Chris  Hale 

Vince  Schmidt 

Job  Aids  Design  and  Effectiveness 

Matt  Waters 

Overview  of  DLA/DAPS 
approach  to  job  aid  design,  its 
successes  and  how  it  is  funded 

Sidebar  A  -  Top  Ten  List 

Editor 

Summary  pulled  from  the 
other  articles  and  author 
recommendations 

Sidebar  B  —  Cognitive  Engineering 
Resources 

Editor 

The  greatest  concern  for  the  issue  is  the  difficult  experienced  with  getting  an  authoritative 
government  representative  to  write  the  demand  side  article  to  start  off  the  issue.  Many  of  the 
people  who  championed  this  work  have  retired  or  moved  on  to  new  positions.  There  are  HSI 
advocates,  but  it  is  difficult  to  find  someone  who  is  able  to  influence  procurements  who  will 
champion  the  cause  of  cognition. 

3.5.4.  Connector 

By  working  to  form  collaborative  alliances  with  competitors  who  were  cognitive  engineering 
specialists  on  the  phase  I  effort  and  across  awardees  on  the  phase  II  effort,  and  because  our  skills 
sets  were  not  competitive  with  theirs,  we  became  viewed  as  a  trusted  entity.  In  addition,  being  a 
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founder  and  co-leader  of  the  HSIWG  put  us  in  connection  with  people  from  the  directors  of  the 
service  HSI  offices  through  experts  in  HSI,  cognitive  engineering  and  systems  engineering 
practice. 

We  purposefully  reached  out  into  the  project  management  community.  We  joined  PMI  and 
attended  local  chapter  meetings,  and  read  project  management  publications  in  order  to 
understand  their  concerns  and  needs. 

This  position  became  an  asset  to  all  the  communities  touched  by  APSE.  Here  are  just  a  few 
examples.  When  Richard  Pew  and  Emily  Roth  sought  systems  engineers  to  review  the  special 
issue  of  CEDM  magazine  on  embedding  cognitive  engineering  into  systems  engineering,  we 
were  able  to  put  them  in  touch  with  systems  engineers  who  were  HSIWG  members.  When  Drs. 
Vince  Schmidt  and  Chris  Hale  were  working  on  human  views  to  insert  into  DoD  enterprise 
architectures  (DODAF),  we  put  them  in  touch  with  NATO  representatives  who  were  defining 
similar  views  for  the  United  Kingdom’s  MODAF  and  with  the  AF  HSI  representatives  who  were 
attending  the  joint  meetings  of  the  two  architecture  groups.  When  Trident  Systems  was  looking 
for  a  cognitive  engineering  house  to  collaborate  on  a  reconfigurable  user  interface,  we  put  them 
in  touch  with  Klein  Associates. 

Our  role  as  connector  enhanced  research,  improved  process  and  extended  practice. 

3.5.5.  I/ITSEC  Booth 

The  I/ITSEC  was  selected  as  a  challenge  event  for  the  project.  I/ITSEC  is  a  training,  education 
and  simulation  event.  Training  is  a  component  of  HSI.  Simulation  is  associated  with  the 
engineering  world.  From  past  experiences  at  the  conference,  we  had  developed  the  opinion  that 
the  EITSEC  exposition  floor  was  a  collection  of  technologies  in  search  of  users. 

We  purchased  booth  space,  designed  and  assembled  a  low-cost  space  and  set  up  APSE  for 
demonstration  and  trial.  The  floor  was  divided  into  sectors  and  personnel  were  sent  out  to  assess 
whether  vendors  were  aware  of  the  relationship  of  their  work  to  HSI  and  how  cognition  was 
incorporated  in  their  development  efforts. 

As  part  of  our  outreach  efforts,  copies  of  the  Insight  HSI  issue  (5)  were  distributed  to  the  vendor 
booths. 

3.6.  Phase  III  Marketing 

Our  concern  was  that  we  could  develop  a  marvelous  tool  that  was  never  exploited.  This  had 
been  the  experience  of  other  sites,  such  as  HSIAC.  At  EITSEC,  we  heard  that  SE  Trace,  another 
tool  developed  for  SBIR  topic  AF05-071,  was  similarly  languishing. 
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We  wanted  to  find  a  way  to  drive  users  to  the  APSE  web  site.  We  conceived  to  target  market 
major  Air  Force  acquisition  program  offices  and  their  supporting  contractors  to  make  them 
aware  of  the  existence  and  value  -  time/cost  saving  and  improved  systems  -  of  APSE. 

Three  Dayton-area  marketing  agencies  were  approached  and  asked  to  provide  estimates  for 
developing  marketing  campaigns.  One  firm  did  not  bid.  A  second  wanted  to  develop  a  web  site 
for  us  -  we  were  not  able  to  communicate  to  them  that  a  web  site  was  what  we  were  marketing. 

A  third  marketer,  one  who  specializes  in  guerilla  marketing,  advised  us  to  find  at  least  one  user 
who  would  employ  APSE  and  provide  testimonials  supporting  the  value  propositions. 

Project  manager  Carl  Pritchard  presented  “The  End  of  Project  Management  as  We  Know  It”  to 
the  Dayton  PMI  chapter  in  August  of  2007.  He  was  contacted  to  obtain  his  inputs  on  our  project 
goals.  After  a  few  minutes,  he  stopped  the  conversation.  He  said  he  was  not  interested  in 
anything  he  had  been  told  about  the  project.  He  advised  we  needed  to  develop  a  two-minute 
video  that  explained  the  problem  and  our  solution  in  layman’s  terms.  We  set  out  to  develop  an 
introductory  video  for  APSE. 

Throughout  the  period  of  perfonnance,  we  took  advantage  of  opportunities  to  demonstrate  APSE 
in  its  early  Backpack  prototype,  CMap  and  PowerPoint  prototype  versions.  These  helped  us  to 
sculpt  the  content  of  the  product  to  user  needs.  They  were  also  part  of  our  marketing  strategy. 
We  used  those  ‘tastes  of  APSE’  as  teasers  to  interest  the  community  in  what  was  to  come. 

3.6.1.  Testimonials 

Dennis  Carlson  demonstrated  the  impact  of  testimonials.  He  videotaped  the  results  of  his  work, 
applied  for  awards,  and  used  prime  contract  project  managers  as  spokespeople  for  the  value  of 
his  work.  We  wanted  to  emulate  his  results. 

Our  contacts  with  Jack  Griffin,  NAVSEA’s  SBIR  manager,  opened  up  an  avenue  to  test  with  the 
systems  engineering  staff  that  support  undersea  warfare.  The  statement  of  work  for  a  CRADA 
was  developed.  It  included  providing  publishable  feedback.  The  Navy  deleted  this  task  from  the 
statement  of  work  because  it  was  against  policy  to  provide  such  testimonials.  The  CRADA  was 
subsequently  abandoned. 

3.6.2.  Video 

In  response  to  project  manager  Carl  Pritchard’s  challenge,  a  script  was  drafted  by  the  design 
team.  Award  winning  videographer  Joanne  Caputo  was  engaged  to  film  the  video.  She 
reviewed  the  script  and  detennined  it  to  be  a  20-30  minute  video  that  would  be  expensive  to 
produce.  Ms.  Caputo  rewrote  the  script.  It  was  bound  at  the  beginning  by  a  silent  segment  in 
which  actors  mimed  the  problem  APSE  was  designed  to  solve  and  at  the  end  by  a  second  silent 
segment  that  demonstrated  the  positive  results  of  APSE  use.  In  between  two  spokespeople,  one 
representing  a  cognitive  engineering,  one  representing  a  human  systems  engineer,  described 
product  attributes,  value  and  uses.  The  final  product  was  approximately  four  minutes  in  length 
in  comparison  to  Mr.  Pritchard’s  challenge  of  a  two-minute  piece. 
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The  movie  was  previewed  at  AHFE  2007  for  the  audience  including  HSI  session  moderator  Dr. 

Marv  Dainhoff,  past  president  of  Human  Factors  and  Ergonomics  Society.  His  response,  “Can  I 

get  a  copy?” 

3.6.3.  Demonstrations 

Demonstrations  were  used  to  validate  APSE  product  requirements.  APSE  was  demonstrated  in 

its  various  forms  in  the  following  venues. 

•  APSE  CMap  version  reviewed  by  Klein  Associates  cognitive  engineers  mid-2007. 

•  A  post-CMap  prototype  was  reviewed  informally  at  INCOSE  International  Symposium  2007 
by  Air  Force  HSI  office  representative  Fran  Greene  and  Booze  Allen  Hamilton’s  Barbara 
Palmer. 

•  Paper  versions  were  presented  to  interviewers  at  Beyond  Phase  II  conference  in  fall  2007. 

•  The  PowerPoint  prototype  was  demonstrated  at  CHEIHMC  workshop  in  October  2007. 

•  Pages  from  the  DotNetNuke  version  were  presented  at  HSI  Working  Group  at  2008  INCOSE 
International  Workshops  in  Albuquerque. 

•  APSE  was  demonstrated  to  Klein  Associates  brownbag  group  in  the  summer  of  2008. 

•  Review  of  APSE  pages  with  cognitive  engineers  Gary  Klein  and  Robert  Hoffman  in  mid 
2008. 

•  Pages  from  the  DotNetNuke  version  were  presented  to  CHI  Systems’  Jennifer  Fowlkes  at 
AHFE  International  in  July  2008.  Showed  Dr.  Fowlkes  the  video  and  described  our  AoA 
activity. 

•  APSE  was  demonstrated  and  made  available  for  test  at  our  EITSEC  2008  booth. 

•  APSE  was  demonstrated  to  Booz  Allen  Hamilton,  Air  Force  HSI  Office  (HSIO)  contractor. 
Ms.  Margaret  Sampson  of  Booz  Allen  Hamilton  was  shown  rough-cut  of  APSE  film, 
introduced  to  deliverables-based  approach  and  software  exercises  including  AoA. 

Each  of  these  interactions  either  affirmed  the  direction  that  was  taken  with  APSE  or  resulted  in 

modifications  or  redesign. 
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4.  Results  and  Discussion 


4.1.  APSE 

4.1.1.  Deliverables 

We  proposed  to  build  an  open-source,  web-enabled  software  application  that  instantiated  a 
model  process  that  embedded  cognitive  systems  into  systems  engineering  practice.  That  product, 
APSE,  is  built  on  the  DotNetNuke  framework  (39).  DotNetNuke  is  an  open  source  web 
application  framework  ideal  for  creating,  deploying  and  managing  interactive  web,  intranet,  and 
extranet  sites  securely.  APSE  fields  are  created  in  an  interface  shell.  Content  is  held  in  an  SQL 
database,  another  open  source  application.  Database  content  populates  the  interface  fields  when 
a  deliverable  is  selected.  Manhour  graphs  are  created  in  real  time  by  Silverlight  2.0,  a  free 
downloadable  tool  provided  by  Microsoft  that  pulls  numbers  from  the  SQL  database. 

APSE  is  free  to  all  registered  users.  There  is  no  fee  for  registration.  Registration  merely  helps 
us  to  track  who  is  using  the  site  and  serves  as  a  security  check  against  hackers. 

APSE  serves  five  target  audiences.  After  login,  the  main  window  comes  up  displaying  a  legend 
(figure  20)  which  defines  the  acronyms  used  for  each  of  the  audiences,  stakeholders  or  users. 
Government  program  managers  represent  system  owners.  Contract  project  managers  represent 
the  contracting  enterprise’s  management  team,  systems  engineers,  human  systems  integrators 
(referred  to  as  human  systems  engineers  in  IEEE  1220  (3))  and  cognitive  systems  engineers  may 


Figure  20:  Target  audience  legend 
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represent  either  the  owner  or  the  contractor  depending  on  the  deliverable  and  context  of  the 
activity. 

The  APSE  main  page  is  shown  in  figure  2 1 .  The  spacious,  streamlined  interface  design  has  been 
termed  “slick”  in  informal  reviews  and  said  to  look  like  a  Star  Trek™  control  panel.  Fields 
dynamically  resize  when  user  click  on  them.  This  is  helpful  for  those  with  vision  degradation 
and  for  occasions  when  the  APSE  is  briefed  before  an  audience. 


System  Development 

Integrated  Master  Plan  /Schedule 


Wednesday,  December  24,  2008 
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The  Integrated  Master  Ran 
flMP)  is  a  contractor- 
prepared  customer  required 
tool  that  is  used  to  track  and 
measure  projecLTask 
accomplishment  It  identdies 
essential  events  program 
milestones  event  entry  and 
exit  criteria  product  reviews 
technical  tasks  and  technical 
development  and  other  nsk 
reduction  activities  The  IMP 
consists  o t  a  table  indexed  to 
the  Work  Breakdown 
Structure  and  a  descnptne 
narrative  ot  execution 
processes  object r/es  and 
control  documents  The  IMP 
is  an  event -based  plan  and 
contractually-binding 
document  that  is  relatively 


Key  Tasks 

GPU  Revew  and  approve  master  pen  contract 
diverse* 

CPU  Ovsrsas  preperaten  of  8tS  and  UP  and  approvs 
Manhours 

ce 


Methods  4  Procedures 

GPU  Ooa^ertf  review 
CPU  Document  renew 
S  E  System  synthesa  Uorytooerdng  Dmelne 

Software  Aids 

GPU  None 

CPU  Ucroeett  Protect 

SE 


Mature  technologies  and  fit  eito  concept 


The  Integrated  Master 
Plan  is  a  contractual 
agreement  between  the 
project  owner  and 
contractor  Work 

desenbed  in  the  MP 
cannot  be  de-scoped 
without  a  formal 
contract  moderation 
Work  not  included  in 
the  IMP  will  not  be 
executed  H  all  system 
elements  -  hardware 
software  and  human  - 
are  to  be  considered 
human  systems 

mteoration  and  coomtrve 


Figure  2 1 :  Sample  APSE  main  page 


Drop  down  menus  at  the  top  allow  users  to  select  from  lists  of  deliverables  (products),  methods 
and  software.  Method  descriptions  are  not  yet  complete.  Software  demonstrations  still  need  to 
be  linked  to  the  drop  down  menu;  they  are  currently  stand-alone  web  pages. 


The  main  page  is  divided  horizontally  into  two  fields.  On  the  left  the  interface  queries  “What  Do 
I  Plan?”  This  is  a  message  to  users  that  the  items  on  the  left  hand  are  those  that  need  to  be 
included  in  a  project  plan  if  cognitive  engineering  is  to  be  successfully  embedded.  On  the  right 
the  interface  asks,  “What  Do  I  Do?”  This  tells  users  that  guides  to  completing  the  deliverable  are 
shown  in  the  fields  above. 


Tabs  on  the  lower  left  represent  the  seven  phases  of  the  model  process  APSE  instantiates. 
JCIDS  has  been  added  as  a  precursor  phase.  Operations  and  Support  and  Retirement,  separate 
phases  in  the  DAS,  have  been  combined.  When  a  phase  is  selected,  a  brief  description  of  the 
phase’s  purpose  appears  in  the  message  bar  at  the  bottom.  Selecting  a  phase  reveals  five 
deliverables  associated  with  that  phase.  Deliverables  may  be  updated  or  repeated  in  more  than 
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one  phase;  they  are  introduced  by  APSE  at  the  earliest  point  in  which  they  are  required  in  the  life 
cycle. 

When  a  deliverable  is  selected,  the  remaining  main  page  fields  are  populated.  In  the  center  is  the 
Product  Summary  which  provides  an  abbreviated  description  of  the  deliverable  and  its  purpose. 
On  the  far  right,  Integration  Opportunity  describes  how  this  deliverable  promotes  collaboration 
and  communication  among  representatives  of  the  target  audiences. 

The  third  column  provides  a  summary  of  the  tasks,  manhours,  methods  and  procedures  and 
software  aids  executed,  expended  or  used  by  each  of  the  stakeholder  groups  in  the  course  of 
completing  the  deliverable.  The  accordioned  windows  expand  when  selected  by  an  APSE  user 
to  reveal  all  the  content  for  a  selected  field  (e.g.,  Key  Tasks)  for  each  of  the  five  stakeholder 
groups. 

The  APSE  deliverables  page  is  represented  in  figure  22.  A  deliverables  page  provides  more 
information  about  the  product  of  interest. 

Manpower  numbers  are  shown  by  default  when  a  deliverables  page  loads.  This  is  because  the 
greatest  burden  to  incorporation  of  cognitive  engineering  in  acquisition  is  the  concern  that  it  will 
overload  a  project  manager’s  budget.  Relative  manhour  estimates  were  created  based  on  the 
author’s  experience  in  project  estimation.  Cognitive  engineering  and  HSI  manhours  were 
provided  or  reviewed  by  cognitive  engineering  practitioners.  Manhour  estimates  are  labeled  as 
To  Be  Reviewed  (TBR)  as  the  numbers  must  be  tailored  to  the  project  at  hand. 
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Figure  22:  Sample  APSE  deliverables  page 
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Input  and  output  conditions  are  listed  upon  selection  in  the  top  field.  These  are  provided  to  help 
with  situating  the  deliverable  in  the  IMP/IMS.  They  are  also  intended  to  support  Earned  Value 
Management  plan  development. 

Figure  22  is  configured  to  show  the  way  in  which  the  interface  expands  a  window  upon 
selection.  In  this  instance,  the  Setting  and  Assumption  fields  are  blown  up.  Setting  describes  the 
acquisition  (project)  state  when  this  deliverable  is  being  generated.  It  is  designed  to  help 
cognitive  engineers  and  human  systems  integrators  place  their  work  in  context  of  the  other 
activities  that  are  simultaneously  occurring.  Assumptions  provides  the  APSE  creator’s  views 
when  they  populated  the  deliverable  page.  Most  often  assumptions  were  made  when  manhour 
estimates  were  made,  so  this  information  supports  review  of  the  APSE  manhour  numbers.  It 
allows  planners  to  understand  what  is  different  in  their  own  situation  from  what  was  assumed  in 
APSE. 

At  the  bottom,  lists  of  tasks,  methods  and  software  aids  are  given.  This  is  in  contrast  to  the  main 
page  which  lists  one  or  at  most  two  tasks,  methods  or  software  for  each  deliverable.  The 
deliverables  page  thus  gives  a  more  complete  picture  of  the  contributions  of  the  represents  of  the 
APSE  target  audiences  for  the  deliverable  in  question. 

APSE  describes  35  deliverables.  Each  is  described  using  pages  similar  to  those  in  figures  21  and 
22.  A  phase- wise  listing  of  the  35  deliverables  is  provided  below. 

Joint  Capabilities  Identification  and  Development 

Operations  Concept 
Task  Analysis 

Functional  [Needs]  Analysis 

Joint  and  Initial  Capabilities  Documents 

Analysis  of  Materiel  /  Non-material  Approaches 

Concept  Refinement 

Cost  (earned  value) 

Systems  Engineering  Plan  with  HSI  Plan 
Analysis  of  [Concept]  Alternatives 

Requirements  (User),  Specifications,  Interface  Control  Documents 
Test  and  Evaluation  Plans 

Technology  Development  and  Demonstration 

Research  and  Development 
System  Concepts 

Modeling  and  Simulation  -  Validating  Against  User  Needs 
Capability  Development  Document 
Infonnation  Support  Plan 
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System  Development 

Prime  Contract 

Integrated  Master  Plan  /  Schedule 

Work  Breakdown  Structure  /  Integrated  Product  Teams  /  Cross  Product  Teams 
Trade  Studies 
User  Interface 

System  Demonstration 

Training 

Developmental  Test  and  Evaluation 
Design  Reviews 
Operational  Assessment 
System  Verification 

Production  and  Deployment 

Analyze  Deficiencies  to  Determine  Corrective  Actions 
Modify  Configuration  to  Correct  Deficiencies 
Pre-Initial  Operational  Capability  Support  Review 
Post-Deployment  Review 
System  Validation 

Operations,  Support  and  Retirement 

Monitor  and  Collect  All  Service  Use  Data 
Analyze  Data  and  Determine  Root  Cause 
Fix  Shortfall  or  Include  in  Next  Increment 
System  Validation 
Retirement 

4,1.2.  Software  Demonstrations 

The  software  demonstrations  were  simulations  of  analyses.  All  looked  at  a  university  intruder 
security  system  consisting  of  a  new  chemistry  lab  classroom  building  with  sensors  and 
communications  systems  and  a  security  call  center.  We  assumed  that  the  chemistry  building  and 
the  call  center  were  to  be  built  simultaneously  and  that  other  buildings  would  later  be  retrofitted 
with  devices  that  could  be  monitored  and  controlled  from  the  call  center. 

Figure  22  shows  the  interface  for  the  Micro  Saint  Sharp  presentation.  The  interface  was 
developed  for  DARPA’s  Rapid  and  Accurate  Idea  Transfer  project.  It  is  based  on  the 
Abstraction-Decomposition  Space  used  in  Cognitive  Work  Analysis.  With  more  abstract 
concepts  at  the  top  and  more  detailed  concepts  at  the  bottom.  In  this  three  column  fonnat,  the 
left  column  provides  general  information  about  the  software  aid.  APSE  designers  provide  the 
purposes  for  the  software  in  the  context  of  embedding  cognitive  systems  into  systems 
engineering  practice.  In  the  left  center  panel,  the  functions  of  the  software  are  listed,  for  the 
most  part  taken  from  vendor  marketing  materials.  At  the  bottom  left  is  a  link  to  the  vendor’s 
product  web  site. 
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At  the  top  right,  the  purposes  of  the  product  demonstration  are  shown.  These  describe  the 
integrative  or  collaborative  attributes  that  can  be  obtained  from  product  use.  In  the  middle  right 
are  the  steps  used  in  the  demonstration.  Analysis  or  design  artifacts  for  each  of  the  steps  are 
used  to  document  the  steps.  Each  of  the  icons  on  the  right  represents  a  link  to  a  pdf,  graphic  or 
flash  movie  file  that  would  appear  in  the  center  window.  The  use  modes  document  is  currently 
displayed.  When  a  user  clicks  on  one  of  the  steps,  the  support  document  is  displayed,  full-screen 
in  the  center  window.  The  window  can  be  reduced  to  its  original  size  if  an  APSE  user  wants  to 
view  other  windows.  In  either  configuration,  scroll  bars  are  supplied  to  support  document 
navigation. 

Manhours  were  collected  for  each  of  the  demonstrations.  The  manhours  are  used  to  populate  the 
graph  at  the  top  of  the  display  where  cost  measures  are  provided  for  planning  purposes.  Below 
the  graph  is  the  binned  retail  price  of  the  software  as  shown  in  figure  23. 
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Figure  23:  Micro  Saint  Sharp  demonstration 

The  bottom  center  window  provides  a  link  to  a  text  document  that  describes  our  experience  with 
using  the  software.  It  notes  glitches  and  resolutions  or  workarounds.  It  describes  features  or 
functions  that  would’ve  improved  usability  or  helped  with  interdisciplinary  communication. 
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A  nine-stop  demonstration  was  conducted  with  Micro  Saint  Sharp.  They  included  creating  a 
two-dimensional  and  a  three-dimensional  behavior  task  network  simulation  of  classroom  use 
based  on  scenarios  documented  in  step  4.  The  task  network  and  two-dimensional  simulation  are 
shown  in  figure  24. 


Figure  24:  Micro  Saint  Sharp  behavior  task  network  (left)  and  associated  2D  animation  (right) 
Figure  25  shows  the  interface  for  the  TestLog  presentation.  The  content  is  laid  out  in  the  same 
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Figure  25:  TestLog  demonstration 


manner  as  the  Micro  Saint  presentation,  general  information  on  the  left.  Specific  information 
about  the  demonstration  is  on  the  right.  Six  test  events  were  created  for  test  log.  They  related  to 
the  build-up  of  the  intruder  call  center.  Cognitive  systems  engineering  methodologies  and  test 
subject  requirements  were  captured  in  the  following  six  test  events. 

1 .  Unit  -  Intruder  speed  estimation  algorithm.  Software  only  test  of  algorithm  that  estimates 
intruder  locomotion  as  function  of  reported  physical  attributes  and  demographics 

2.  Component  —  Location  of  intruder  section  of  situation  awareness  user  interface.  Is  assumed 
to  integrate  four  algorithms  to  provide  users  with  25  percent,  50  percent  and  90  percent  error 
probability  circles.  Intruder  speed  estimation  algorithm  is  part  of  this  suite.  Because  the  location 
function  is  considered  a  critical  function  of  the  system,  a  table  top  analysis  is  executed  after  the 
test  event  so  subject  matter  experts  can  determine  whether  the  design  approach  is  adequate,  if  it 
incorporates  the  latest  understanding,  and  if  it  is  appropriate  for  both  experts  and  novices. 

3.  Pilot  -  situation  awareness  user  interface  clickable  prototype.  Test  plan  assumes  that  enough 
of  the  user  interface  exists  to  execute  usability  tests. 
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4.  Subsystem  (integration  event  -  situation  awareness  user  interface,  delivered  product).  One  of 
the  four  call  center  workstations  is  tested  in  its  fully  integrated,  delivery  configuration. 

5 .  System  -  call  center.  This  is  the  fully  integrated  system  undergoing  verification  testing  prior 
to  delivery.  A  second  table  top  analysis  is  incorporated  to  firm  up  training  requirements  and 
identify  any  late -phase,  minor  changes  that  could  be  incorporated  into  the  hardware  and 
software  that  would,  when  traded  against  training  costs,  save  the  program  money. 

6.  Acceptance  -  lives  saved.  This  is  the  validation  event  conducted  as  part  of  a  simulated 
emergency  event. 

Two  examples  of  the  table  top  analysis  method  was  generated  with  example  questions  and 
simulated  output  tables.  The  first  analysis  was  used  as  part  of  requirements  validation  to  ensure 
that  the  design  approach  was  satisfactory.  The  second  was  conducted  as  part  of  system  test  in 
order  to  capturing  training  requirements. 

TestLog  exports  its  documents  as  html  files.  This  eased  construction  of  the  demonstration 
interface.  Artifacts  associated  with  each  of  the  eight  steps  in  the  center  right  panel  are  html  files 
created  from  within  TestLog.  This  illustrates  the  flexibility  of  the  presentation  interface  for 
APSE  purposes.  No  matter  the  file  type,  this  interface  can  accommodate  it. 


Task  Architect,  a  bookkeeping  aid  for  task  analysis  results,  was  similarly  demonstrated.  Thirty-six 
task  parameters  were  populated  for  two  selected  tasks.  Tasks  were  taken  from  the  University 
Intruder  scenario  that  was  developed  in  conjunction  with  the  Micro  Saint  Sharp  demonstration.  Task 
for  a  four-station  call  center  were  analyzed. 

4,1.3.  Analysis  of  Alternatives 

Ordered  Phoenix  Integration  Software  for  Analysis  of  Alternatives  demonstration  exercise 
After  completing  work  on  Task  Architect  and  TestLog  (anticipated  end  of  April),  the  design 
team  will  assemble  a  suite  of  models  to  perform  an  Analysis  of  Alternatives  related  to  the 
University  Intruder  scenario.  We  have  discussed  our  initial  plan  which  will  include  Excel 
models,  perhaps  some  of  the  other  tools  we  have  already  exercised,  for  example  Micro  Saint 
Sharp,  and  a  cost  model.  This  exercise  will  help  to  bring  in  measurable  cognitive  systems 
engineering  attributes  and  provide  lessons  in  linking  cognitive  systems  engineering  and  systems 
engineering  in  this  key  acquisition  activity. 

Steps  1-5  have  been  previously  completed.  We  are  simultaneously  working  on  steps  6  and  7.  A 
architecture  of  models  has  been  developed  which  includes  10  integrating  models,  one  for  each 
Measure  of  Effectiveness.  A  decision  model,  the  Integration  Model,  will  aggregate  the  results  of 
each  of  the  10  integrating  models  in  a  method  similar  to  QFD  in  order  to  deliver  a  “score”  for 
each  of  the  two  alternatives  we  generated  in  the  previous  reporting  period. 

Ten  Integrating  Models  /  MOEs 

•  Lethality  Index 

•  Time  to  Resolve  and  Report 
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•  Safety  Index 

•  Accuracy 

•  Affordability 

•  Availability 

•  Preparedness 

•  Positive  Affect 

•  Refurbishment  Efficiency 

•  Disposal  Efficiency 

Supporting  the  10  integrating  models,  are  26  contributing  models.  These  were  identified  with 
documented  traceability  to  steps  1-3.  We  do  not  plan  to  populate  all  26  of  these  contributing 
models.  These  would  be  required  if  the  full  Analysis  of  Alternatives  described  in  the  exercise 
were  to  be  executed.  Our  intention  is  to  simulate  an  Analysis  of  Alternatives  that  includes 
human  considerations  -  how  it  would  be  constructed  and  executed.  We  plan  to  populate  only 
those  models  that  are  necessary  to  achieve  this  demonstration. 

The  primary  goal  of  the  Call  Center  is  to  reduce  the  lethality  of  an  intruder  incident.  The 
primary  MOE  for  this  is  an  algorithm  we  call  “Lethality  Index.”  It  is  dependent  upon  attributes 
of  the  weapon,  the  building  design,  and  the  ability  of  Call  Center  personnel  to  manage  the 
situation  through  manipulation  and  communication.  In  describing  the  model,  we  have  defined 
parameters,  identified  their  purpose  in  the  analysis,  described  the  calculation  method  and 
identified  contributors,  such  as  a  weapons  database. 

A  similar  approach  was  taken  for  a  “Time  To  Resolve  and  Report  [Intruder  Incident]”  model.  In 
order  to  develop  this  timeline  model,  an  analysis  scenario  was  created.  This  scenario  is  neutral 
to  the  two  alternatives  being  analyzed.  It  describes  observable  intruder  actions  and  goals. 

The  timeline  model  motivates  a  simulated  cognitive  walkthrough  of  storyboards.  Staffing 
profiles  were  determined  as  part  of  alternative  definitions.  A  database  of  individual  and  team 
cognition  results  is  being  generated.  Statistics  from  this  database,  the  User  Interface  Test  Model, 
will  feed  into  the  integrating  Accuracy  model  as  well  as  the  Time  to  Resolve  model. 

Simultaneous  with  the  User  Interface  Test  Model  a  Manpower,  Personnel  and  Workload  model 
is  being  developed.  Individual  tasks  taken  from  the  timeline  are  aggregated  into  team  tasks. 
Roles,  responsibilities,  skills,  knowledge  and  aptitudes  are  defined.  A  human  capital  objects 
model  is  used  to  generate  a  proficiency  rating.  When  the  proficiency  rating  number  is  compared 
with  incumbents  or  the  target  population,  this  enables  an  estimate  of  compensation  required  to 
hire  a  person  with  this  proficiency  or  the  investment  in  training  required  to  grow  a  person  with 
this  proficiency  from  the  pool  of  qualified  incumbents.  Thus,  this  impacts  the  Affordability 
MOE  in  addition  to  Accuracy. 

A  walkthrough  of  progress  was  conducted  with  Klein  Associates  personnel.  Suggestions  for 
improvement  were  made.  Subjectively  the  feedback  was  this  was  a  very  exciting  approach.  All 
parties  wished  this  exercise  had  commenced  at  the  beginning  of  Phase  II  as  there  has  been  a 
great  deal  of  learning  from  it  already. 
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4.2.  Market  Development,  Awareness  and  Education 

Meetings  were  planned,  lead  and  attended.  Papers  were  written  and  presented.  The  HSIWG  was 
founded,  organized,  documented  and  energized.  Cross-disciplinary  and  cross-organizational 
connections  were  made.  These  activities  were  sometimes  referred  to  as  marketing,  sometimes  as 
social  engineering. 

We  succeeded  to  some  extent  in  getting  practitioners  and  organizations  to  look  beyond 
themselves.  Members  of  INCOSE,  IEEE  and  HFES  were  interacting  and  exchanging  ideas. 

They  were  sharing  plans.  An  MOU  between  INCOSE  and  HFES  was  initiated  and  is  in  review. 

The  IEEE  papers  (3)  (29)  to  which  we  contributed  pushed  reached  audiences  with  messages  that 
were  elementary  to  systems  engineers,  but  were  considered  paradigm  shifters  by  the  cognitive 
engineering  people. 

We  demonstrated  that  deliverables,  not  process,  represented  points  of  integration.  People  do  not 
work  at  the  process  level.  Practitioners,  even  when  they  are  only  systems  and  specialty 
engineers,  interact  on  the  products  they  must  produce  together.  This  philosophy  was  shared  with 
Dr.  Gavan  Lintern  during  his  development  of  a  Cognitive  Systems  Engineering  -  Systems 
Integration  workshop. 

Dr.  Lintem  was  invited  to  give  a  workshop  to  South  African  systems  engineers.  He  asked  that  we 
review  the  approach.  He  had  intended  to  present  at  the  process  level,  but  was  convinced  to 
address  the  integration  at  the  product  level  following  the  APSE  approach.  His  acceptance  of  this 
approach  enabled  us  to  suggest  incorporation  of  the  lessons  and  formative  suggestions  that  were 
developed  as  part  of  this  contract.  The  workshop  was  very  well  received,  and  Dr.  Lintern  plans 
to  seek  other  opportunities  to  give  it. 

4.2.1.  HSIWG 

Starting  from  two  members  in  January  2006,  the  organization  has  grown  to  over  150.  The 
bylaws  drafted  by  this  report’s  author  in  February  2008  and  adopted  by  membership  in  March 
included  provision  for  people  from  organizations  outside  INCOSE  to  participate  in  the  group  as 
adjunct  members.  This  clause  laid  the  foundation  for  adoption  of  a  policy  similar  to  that  of 
HFES.  For  a  nominal  annual  fee  (<  $20),  anyone  may  join  and  participate  in  an  HFES  technical 
group.  This  is  a  mechanism  for  breaking  down  the  walls  between  specialties  that  are  barriers  to 
needed  solutions. 

INCOSE  is  on  record  as  stating  that  humans  must  be  treated  within  the  system  design  boundary, 
that  HSI  is  an  integral  part  of  systems  engineering.  Some  INCOSE  members  have  stated  they 
believed  HSI  would  BE  systems  engineering  in  another  10  to  20  years. 

The  HSIWG  is  working  with  the  requirements  group  to  establish  guidelines  for  the  development 
of  HSI  requirements.  At  the  Assistant  Secretary  of  Defense  levels,  standards  for  HSI  practice  are 
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being  developed.  These  are  useful,  but  requirements  statements  and  RFP  language  represent 
contractual  comments  that  must  be  satisfied.  Requirements  the  vehicle  by  which  system 
attributes,  which  include  human  contributions  and  constraints,  will  fulfill  mission  needs,  address 
capability  gaps  and  make  fielding  these  systems  more  affordable  over  their  operational  life. 

The  Hoffman-Deal  paper  (29)  made  clear  the  differences  between  infonning  design  and 
specifying  design.  The  former  is  not  enforceable;  it  has  no  teeth.  APSE’  pages  on  configuration 
management  explain  the  process  for  modifying  requirements  and  specifications  during  a 
program’s  execution.  Together,  this  infonnation  should  help  cognitive  engineers  to  impact 
designs  with  their  work  products  and  to  understand  how  the  findings  from  continuous  learning 
can  be  incorporated  in  the  development  process. 

Shall  statements  are  not  the  only  way  in  which  a  system  can  be  specified.  HSIWG  is  also 
working  with  the  Model-Based  Systems  Engineering  (MBSE)  Working  Group,  individuals  who 
believe  dynamic  models  of  systems  are  more  fruitful  and  accurate  ways  of  delimiting  system 
features.  This  relationship  places  new  demands  on  cognitive  engineering  and  HSI  community. 

The  MBSE  community  seeks  cognitively  accurate  models  of  agents  placed  in  their  systems 
models.  They  would  like  to  answer  the  question,  “What  happens  when  you  put  160  colonels 
inside  a  one-acre  area?”  as  is  done  with  an  air  operations  center.  Personality  types,  conation,  and 
affect  will  need  to  be  incorporated  in  these  models.  Advocates  of  MBSE  are  communicating 
their  needs;  members  of  the  cognitive  engineering  community  have  said  no  one  can  do  that 
today.  It  remains  to  be  seen  whether  they  will  be  able  to  do  that  tomorrow. 

The  benefits  of  the  collaborative  HSIWG  aside,  it  may  be  approaching  a  leadership  crisis. 
Chairing  a  diverse  group  of  that  size  is  enormously  demanding.  Few  professionals  feel  they  have 
the  time  to  do  the  job  properly.  Elections  to  be  held  this  April,  may  determine  whether  the  group 
will  continue. 

4.2.2.  I/ITSEC  Booth 

I/ITSEC  entertained  about  16,000  visitors  and  over  400  exhibitors.  Our  goal  was  to  set  up  a  very 
humane  booth  in  the  midst  of  electronic  overload.  Our  neighbors  boasted  multiple,  huge  video 
displays,  large  simulators,  models,  alcohol,  explosions  and  target  shooting  which,  if  separated, 
would’ve  been  manageable. 

Figure  26  shows  the  APSE  booth.  Two  computers  enabled  us  to  give  product  demonstrations  or 
for  people  to  try  APSE  for  themselves.  None  of  the  visitors  chose  the  latter.  People  who  did 
stop  preferred  us  to  demonstrate  the  tool.  The  promotional  video  was  run  continuously  to  attract 
the  attention  of  passers  by.  An  easel  containing  a  flip  chart  held  many  different  messages  during 
the  conference.  The  contents  of  the  message  changed  based  on  engagements  and  discussions 
with  booth  visitors.  Our  plan  was  to  look  like  a  library  with  comfy  chairs  in  which  people  could 
rest  and  review  the  product.  Few  took  the  opportunity  to  rest. 

Many  people  walked  by  our  booth  without  a  second  look.  A  booth  designer  came  by  just  to  look 
at  it  and  marvel  at  the  simplicity  and  attractiveness  of  the  design.  Several  very  enthusiastic 
people  stopped  and  talked  for  extended  periods.  Those  who  stopped  with  interest  were  shown  a 
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demonstration  of  the  product.  For  example,  representatives  of  SPAWAR  were  very  interested  in 
APSE,  said  they  had  done  something  similar  but  that  it  didn’t  have  nearly  as  good  an  interface  as 
APSE. 


Figure  26:  The  APSE  booth  at  I/ITSEC 


Below,  in  Table  10,  is  a  list  of  individuals  who  either  stopped  by  the  booth  with  interest  in  HSI 
or  CSE  or  in  APSE  particularly  or  who  were  engaged  during  walkarounds  of  the  floor. 

During  walkarounds  Deal  Corp  personnel  engaged  vendors  in  discussions  about  how  they  fit  into 
the  human  systems  integration  framework,  their  incorporation  of  cognitive  attention  and 
decision-making  principles  and  cognitive  systems  engineering  and  their  use  of  task  analyses  in 
defining  training  or  other  product  requirements.  Most  of  those  engaged  were  not  aware  that 
Training  was  a  component  of  human  systems  integration.  They  accepted  requirements  in  an 
“over- the- wall”  fashion  saying,  ‘Give  us  your  requirements  and  we’ll  apply  our  technology  to 
developing  training  products.’  Prior  to  the  conference,  we  assumed  we  would  encounter  a  block 
of  technology  that  was  divorced  from  the  needs  of  the  end  user.  That  was  also  our  conclusion  as 
we  left  I/ITSEC. 

In  addition  to  the  booth  and  floor  walks,  we  networked  with  people  at  lunch  and  between 
presentations.  The  idea  was  to  make  people  aware  of  human  systems  integration,  cognitive 
engineering  and  APSE.  We  had  calling  cards  printed  with  the  product  name,  purpose  and  URL. 
Deal  Corp  was  not  promoted  on  the  calling  card  in  keeping  with  our  efforts  to  avoid  branding 
product  for  fear  that  a  “not  invented  here”  syndrome  would  reduce  its  use. 
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Table  10:  Booth  Visitors  or  Other  Engaged  During  Floor  Walks 


Name  (last, fir st)/Title 

Affiliation 

Bryant,  Chris 

Director 

Human  Systems  Integration 

Sys  Technologies 

Catlett,  David 

Director 

Center  for  Trans  formative  Research 

DeBargis 

Senior  Manager 

Lockheed  Martin  Canada 

Decker,  William  M. 

Director,  Technology  Transition 

Learning  Center  of  Excellence 

Defense  Acquisition  University 

South  Region 

Goodman,  Michael  S. 

General  Dynamics  C4  Systems 

Gordon,  Doretta  E.  Ph.D. 

Acting  Director 

Emerging  Training  &  Perfonnance 

Technologies  Department 

Training,  Simulation  and  Perfonnance 
Improvement  Division 

Southwest  Research  Institute 

Green,  Olin  S.,  Jr. 

Sr.  Architect 

Defense  Technologies 

Kratos,  Defense  and  Security  Solutions 

Kauchak,  Marty 

Editor 

Military  Training  Technology 

KMI  Media  Group 

Sampson,  Thomas,  LT 

Asst.  Fleet  Aviation  Training  Systems 

Chief  of  Naval  Operations,  N882B2A 

Smith,  Eddie  B. 

RAVLLC 

Steinman,  Jeffrey  S. 

Ph.D. 

President  &  CEO 

WarpIV  Technologies,  Inc. 

Tubell,  Wally 

Professor  of  Engineering 

Engineering  and  Technology  Department 

Defense  Acquisition  University 

South  Region 

Walrond,  Col  Thomas 

United  States  Air  Force 

Joint  Forces  Command 

Joint  Warfighting  Center 

Waters,  Matt 

JPA  Program  Lead  Contractor 

DLA/DAPS 

Whitted,  Gary  A. 

Senior  Systems  Engineer 

Systems  Engineering  Operations 

Ball  Aerospace  &  Technologies  Corp 
Systems  Engineering  Services 
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Subsequent  to  the  conference,  several  people  registered  for  use  of  the  APSE  web  application. 

We  also  notified  people  from  previous  meetings  of  the  availability  of  the  tool  and  garnered  a  few 
additional  users.  At  last  count,  however,  there  were  still  less  than  a  dozen  registered  users.  The 
concern  we  had  about  the  site  languishing  remains  a  concern  and  something  that  needs  to  be 
worked  after  the  contract’s  conclusion. 

4.3.  Phase  III  Marketing 

4.3.1.  Testimonials 

We  were  not  able  to  obtain  testimonials  about  APSE’  effectiveness  prior  to  the  end  of  the  period 
of  performance.  In  part,  this  is  because  APSE  pages  were  being  populated  as  late  as  the  end  of 
November  2008.  By  then,  the  team  was  focusing  on  I/ITSEC  preparations  and  time  did  not 
permit  us  to  seek  a  user  who  was  willing  to  test  the  product.  Kelly  Neville,  a  professor  at 
Emery-Riddle  University  has  said  she  will  use  APSE  if  she  winds  up  again  teaching  the  system 
development  course  she  taught  in  2008.  If  successful,  this  could  provide  one  statement. 

We  still  feel  that  testimonials  will  differentiate  APSE  from  useful  sites  that  have  languished  in 
the  past.  It  is  difficult  thing  to  manage  when  developing  products  for  government  use,  as 
government  employees  are  not  permitted  to  even  appear  to  endorse  products.  A  DoD  prime 
contractor  could  contribute  what  is  needed  if  a  trial  can  be  arranged  and  APSE  provides  value. 

4.3.2.  Video 

Figure  27  shows  the  opening  and  closing  messages  of  the  APSE  movie.  The  problem  on  the  left, 
is  that  customers  (users)  hate  the  new  product,  the  system.  They  refuse  to  use  it,  it  is  costing  the 


Figure  27:  APSE  movie  beginning  and  ending 
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owner  a  fortune.  At  the  end,  by  using  APSE,  designers  have  better  incorporated  user  and  owner 
needs.  They  have  a  better  handle  on  the  total  cost  picture. 

The  spokeswoman  for  the  cognitive  engineering  discipline  describes  the  how  APSE  helps  to 
address  the  cognitive  requirements  of  work  (figure  28)  -  critical  decisions,  managing  uncertainty, 
planning  and  re-planning,  sense  making  and  problem  detection. 


Figure  28:  Illustrating  the  cognitive  requirements  of  work 


The  video  introduces  people  to  the  functions  of  APSE  when  they  navigate  to  the  web  site.  We 
also  plan  to  post  it  to  You  Tube  as  another  avenue  to  drive  people  to  the  web  site.  The  humorous 
ending  is  something  that  will  intrigue  the  upcoming  generation  of  managers  and  engineers. 

In  retrospect,  it  might  have  been  advisable  to  generate  a  similarly  entertaining  video  that  more 
generally  describes  the  problems  that  arise  when  human  needs  are  excluded  and  the  solutions 
that  HSI  and  cognitive  engineering  offer  as  opposed  to  one  that  was  specifically  for  APSE.  A 
more  general  treatment  could’ve  become  a  sales  tool  for  HSI  practitioners,  useful  for 
demonstrating  to  engineers  and  managers  the  value  human  system  engineering  provides. 

4.3.3.  Demonstrations 

The  demonstrations  were  a  non-invasive,  safe  approach  to  validating  our  process  and  product 
requirements.  They  very  effectively  helped  us  to  arrive  at  an  attractive  presentation  and  useful 
content. 

After  the  demonstration  to  Booz  Allen  Hamilton’s  Margaret  Sampson,  she  noted  that  the  APSE 
project  addressed  the  following  areas  identified  as  critical  by  the  Air  Force  HSIO. 

•  Selection  of  important  points  of  intersection  -  HSIO  has  been  working  to  identify  the  places 
in  the  acquisition  process  where  HSI  analysts  and  engineers  should  interact.  APSE  team  has 
identified  these  with  their  deliverables  list. 
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•  Software  tools  linking  systems  engineering  to  HSI  -  HSIO  adopting  a  survey  and  selection 
approach.  APSE  exercises  and  demonstrates  how  viable,  existing  tools  can  used  to  make  the 
connection.  Sampson  acknowledged  this  to  be  an  efficient,  effective  approach. 

•  Analysis  of  Alternatives  Approach  -  incorporation  of  human  attributes  in  an  AoA  has  proven 
elusive.  APSE  team  believes  human  considerations  can  be  included,  but  the  data  preparation 
stage  of  the  AoA  process  will  differ  from  that  used  by  technologists. 

As  stated  in  section  3,  we  do  not  believe  the  survey  and  selection  approach  to  software  selection 
will  be  useful.  Without  exercising  the  software,  it  is  impossible  to  detennine  its  effectiveness. 
Additionally,  software  products,  which  by  their  very  simplicity  are  overlooked  by  surveys, 
demonstrate  remarkable  capabilities  for  linking  systems  engineering  and  HSI.  Displays  of  step- 
by-step  instances  of  software  implementation  help  users  to  envision  how  those  products  can  meet 
acquisition  practitioner  needs. 

We  did  not  complete  our  AoA  exercise.  However,  after  reviewing  APSE  progress,  particularly 
analysis  of  alternatives  and  APSE  video  with  CHI  Systems’  Dr.  Jennifer  Fowlkes,  she  observed 
“Even  if  you  don’t  finish  the  analysis  of  alternatives,  just  getting  through  [step  three]  the 
Measures  of  Effectiveness  is  amazingly  valuable.”  We  certainly  achieved  that  and  more  by 
proceeding  to  step  six,  model  construction. 
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5.  Conclusions 

APSE  instantiates  a  process  for  embedding  cognitive  systems  into  systems  engineering  practice 
which  is  built  upon  the  Defense  Acquisition  System.  As  such,  it  meets  the  topic  requirement  of 
being  acceptable  to  the  acquisition  community. 

There  were  some  indications  that  high-powered  software  for  designing  user  interfaces  was 
desired  from  the  contractors.  Software  exists  for  developing  user  interfaces.  Other  techniques, 
such  as  providing  XML  language  so  users  can  tailor  their  own  interfaces,  are  in  hand  or  in 
progress.  These  do  not  solve  the  problem  of  linking  the  interfaces  to  user  needs.  Additionally, 
configuration  control  would  be  difficult  with  such  tools  and  could  create  vulnerabilities 
particularly  in  a  system-of-systems  environment. 

Cognitive  engineering  researchers  at  the  CHEihmc  meeting  in  October  2007  sought  a  definition 
of  human  systems  integration  that  would  allow  their  research  to  flourish  and  to  be  incorporated 
in  system  design.  This  does  not  recognize  the  realities  of  the  current  environment.  It  is  an 
approach  that  does  not  provide  a  path  to  incorporation  of  cognitive  engineering  in  systems 
engineering  practice.  It  could  be  advantageously  disruptive,  but  it  would  be  difficult  to 
implement. 

During  meetings  at  Glen  Helen  in  phase  I  (20),  the  desire  to  remove  the  artistry  from  cognitive 
engineering  was  expressed.  It  is  our  opinion  that  the  opposite  course  will  be  more  profitable. 

We  need  to  accept  the  challenge  of  building  better  artists.  That  suggests  an  institutionalized 
educational  solution,  but  that  approach  is  not  necessarily  the  answer  either.  Universities  are  not 
contrived  to  provide  interdisciplinary  education.  At  MIT,  the  aerospace  engineering  curriculum 
includes  a  course  called  “Unified  Engineering.”  At  one  time,  the  course  work  of  fluid  dynamics, 
mechanics,  structures,  etc.  were  conceived  to  be  concurrently  taught  and  integrative  problems 
sets  assigned.  In  practice,  each  discipline  was  taught  independent  of  the  others.  Problems  sets 
were  populated  with  exercises  specific  to  each  field. 

During  the  2008  HSIWG  meeting,  integrative  college  curriculums  were  reviewed.  We  observed 
a  similar  approach  being  taken.  The  necessary  topics  were  individually  present,  but  the  means  to 
bring  them  together  was  lacking. 

APSE  takes  an  approach  that  has  been  shown  by  Defense  Acquisition  University  to  satisfy  user 
needs.  Continuous  learning  modules  provide  information  required  to  do  the  job  at  end  when  that 
information  is  needed  -  just-in-time  education.  This  tactic  can  reach  people  who  are  developing, 
operating  and  sustaining  systems  today.  We  may  not  have  to  wait  until  a  generation  of  properly 
trained  HSI,  systems  engineering  and  cognitive  engineering  becomes  available. 

That  being  said,  it  was  asserted  by  Dr.  Robert  Hoffman  that  the  current  generation  of  acquisition 
specialists  must  pass  before  meaningful  change  can  happen.  It  is  our  observation  that  the  same 
can  be  said  for  the  current  generation  of  cognitive  engineers.  The  founders  are  bound  to  their 
research  backgrounds.  They  enjoy  theoretical  discussions  and  debate.  Their  productivity  is 
measured  in  published  papers.  As  has  been  discussed,  these  individuals  are  not  able  to  directly 
influence  design  because  technical  management  activities  are  not  part  of  their  practice.  They 
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can’t  write  specifications  that  translate  their  products  into  something  that  is  useful  for  engineers 
and  manufacturers. 

There  is  a  second  generation  of  cognitive  engineers  in  the  field  at  this  time.  They  are  the  ones 
who  have  been  tasked  with  field  work  and  interacting  with  the  engineering  team.  These 
individuals,  and  we’re  proud  to  include  them  as  contributors  here,  are  experiencing  the  tempo, 
procedures,  and  constraints  of  a  project.  They  are  adapting  techniques,  bom  of  research,  to 
satisfy  the  driving  need  to  get  the  product  out  the  door.  They  are,  to  use  Wayne  Zachary’s 
phrase,  able  to  answer  the  mail. 

One  thing  they’ve  discovered  is  that  there  is  an  entity  between  cognitive  engineers  and  software 
developers  that  is  missing.  That  person  is  a  human-computer  interface  designer  or  a  graphic 
designer.  These  people  are  required  to  make  the  translation  between  analysis  and  production. 

We  discovered  this  omission  when  developing  the  APSE  interface  and  supplied  it  by  hiring  a 
graphic  designer.  In  retrospect,  the  inclusion  of  these  disciplines  would’ve  enriched  the  results 
of  this  endeavor. 

This  document  is  the  final  report  for  a  project.  The  challenge  presented  was  not  treated  solely  as 
a  project.  Shortfalls  in  addressing  cognitive  work  negatively  impacts  core  Air  Force  Missions  — 
air  combat,  air  mobility  command,  control,  communications,  computers/intelligence 
surveillance,  and  reconnaissance,  and  information  warfare  and  space  operations.  This  challenge 
is  at  least  50  years  old.  It  has  pennutations  that  affect  artificial  intelligence  and  intelligent  agent 
design,  automation  and  modeling  and  simulation  to  name  a  few.  We  assumed  the  Air  Force 
wanted  the  problem  solved,  and  we  undertook  to  retire  it. 

Our  approach  was  an  aggressive  one.  We  combined  work  outside  the  contract  with  elements  in 
our  work  statement  and  the  results  of  each  to  advance  both.  APSE  does  is  not  all  that  it  could  be. 
There  are  things  we’d  intended  to  include  that  were  not  achieved.  As  Margaret  Sampson 
attested,  APSE  addressed  some  of  the  most  difficult  issues  preventing  integrated  HSI  practice  - 
integration  points,  software  and  the  Analysis  of  Alternatives. 

We  believe  we  have  benefited  the  systems  engineering,  cognitive  engineering,  human  systems 
integration  and  program/project  management  fields  substantially.  When  we  started  there  was 
little  or  no  recognition  of  the  importance  of  human  systems  integration  and  cognitive  engineering 
that  would  let  alone  a  market  for  new  software  products.  The  cognitive  engineering  and  human 
systems  integration  practitioners  had  the  tools  they  believed  they  needed.  The  systems  engineers 
didn’t  know  they  needed  to  more  than  they  already  were  doing.  We  believe  this  activity  has 
begun  to  open  the  market  for  new  products  and  approaches  by  working  across  disciplinary 
boundaries  and  encouraging  others  to  do  the  same.  We  are  satisfied  and  gratified  by  the  value 
we  believe  we’ve  delivered  to  the  Air  Force,  to  the  Department  of  Defense  and  to  people  who  are 
on  the  receiving  end  -  users,  sustainers  and  owners  -  of  the  products  we  build. 
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6.  Recommendations 

6.1.  APSE 

APSE  is  finished,  but  it  is  not  complete.  We  achieved  a  rudimentary  capability  that  will  serve  to 
bring  systems  engineering  and  cognitive  engineering  together.  It  could  be  much  more  than  it  is. 
We  recommend 


1)  The  Analysis  of  Alternatives  exercise  be  continued  to  completion.  It  is  on  the  brink  of 
delivering  the  value  required  by  the  Air  Force  HSIO  office.  The  topic  itself  contains 
several  publishable  topics  not  only  on  AoA  but  also  on  the  design  of  emergency  response 
systems. 

2)  Additional  software  demonstrations  be  conducted.  These  have  many  advantages  over  the 
collect  and  collate  methods  that  have  been  implemented  in  the  past  to  little  effect. 
Additionally,  the  “our  experience”  sections  could  result  in  software  improvements. 
Arranging  for  a  controlled  demonstration  of  APSE.  This  would  enable  further  refinement 
of  the  tool’s  usability  and  its  content.  Additionally,  it  would  provide  the  testimonials 
needed  to  move  it  from  a  research  project  into  a  useful  tool  for  the  field.  It  would  also  be 
useful  to  refine  and  validate  the  manhour  estimates  in  the  tool.  Reopening 
communication  with  people  contacted  at  the  beyond  Phase  II  conference  could  create  an 
appropriate  venue. 

3)  Continue  marketing  APSE.  Take  it  INCOSE  Workshop  and  2009  Symposium,  HSIS 
2009,  HFES  2009  and  the  PMI  north  American  congress.  This  would  not  only  increase 
usage  numbers,  but  it  allow  advocacy  for  HSI  and  cognitive  engineering  to  continue. 

4)  Engaging  Defense  Acquisition  University.  A  link  to  APSE  is  a  first  entry  to  making  the 
information  available  to  acquisition  professionals.  APSE  could  also  be  converted  into  a 
continuous  learning  module  that  would  supplement  the  current  e-leaming  content  on  the 
site.  The  university  does  not  have  funds  to  develop  continuous  learning  modules,  but  will 
guide  the  development  of  modules  if  outside  funding  is  provided. 

5)  Work  to  complete  a  validated  cost  model.  We  had  planned  to  meet  with  DoD  cost 
estimates  and  reviewers  to  define  model  data  requirements,  algorithms  and  data 
collection  process.  CHI  Systems  coordinated  a  panel  at  HFES  2008  to  discuss  the  topic. 
This  raised  awareness  in  that  community,  but  the  challenges  of  developing  a  model  that 
acceptable  to  acquisition  professionals  will  take  more  than  a  discussion  among  people 
unfamiliar  with  cost  modeling  if  results  are  to  be  achieved. 

6)  Modify  the  tool  to  accept  additional  deliverables.  Thirty-five  products  were  selected 
because  our  findings  showed  them  to  be  the  most  influential.  Cognitive  engineers  and 
human  systems  integrators  influence  other  products  as  well.  It  would  be  beneficial  to 
modify  the  product  so  it  could  be  extended. 
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6.2.  APSE  version  2.0 

If  continued  marketing  shows  a  demand  for  APSE  and  the  involved  communities  of  practitioners 
continue  to  come  together  to  define  a  joint  practice,  then  there  will  be  a  need  for  software  that 
supports  their  integration.  We  suggest  that  a  team  that  included  project  managers,  system 
engineers,  IT  architects,  human  systems  integrators,  cognitive  engineers,  human  computer 
interface  designers,  graphic  designers,  software  engineers,  display  developers,  and  cost  modelers 
would  be  equipped  to  develop  this  high-power  support. 

This  activity  would  include  improving  upon  and  confederating  the  software  that  was 
demonstrated  during  this  project.  We  found  there  to  be  a  great  deal  of  room  for  improvement. 

6.3.  The  Human  Performance  Discipline 

Practice  within  the  human  performance  discipline  could  be  strengthened  by  addressing  needs 
identified  during  this  project.  We  recommend: 

1)  Extend  work  on  affect  and  conation.  As  mentioned  these  features  are  being  demanded  by 
the  MBSE  community.  This  would  include  personality  studies  and  typing.  Simulation 
agents  that  respond  based  on  emotion  and  that  can  be  motivated  or  discouraged  are 
desired.  Incorporation  of  these  will  also  support  development  of  aids  to  political, 
economic,  social,  information  and  intelligence  operations.  Advances  would  help  with 
threat  characterization. 

2)  Eliminate  the  notion  of  domains  in  the  definition  of  HSI.  At  one  time  ergonomics  was 
defined  to  incorporate  all  aspects  of  the  human  experience.  Over  time  this  was  narrowed 
and  human  factors  was  introduced  to  encapsulate  them  all.  It,  too,  was  subject  to 
specialization  and  HI  was  coined. 

Human  engineering,  whatever  it  is  called,  is  an  inherently  integrative  process.  Dennis 
Carlson,  the  most  effective  practitioner  we  observed,  does  not  think  in  compartments,  he 
envisions  the  experience  of  use  and  practices  with  the  artifacts  using  physical  prototypes 
to  assure  that  mission  and  ownership  goals  are  achieved. 

We  recommend  that  physiology,  medicine,  and  medical  delivery  be  included  when  HSI  is 
considered.  This  will  be  difficult  to  do  so  long  as  HSI  is  regarded  organizationally  rather 
than  as  a  process  that  is  implemented  in  practice. 

3)  Replace  the  individuals  who  were  champions  of  cognitive  engineering.  These  projects 
have  created  a  momentum  that  will  help  the  Air  Force  to  address  critical  missions. 
Infonnation  intensity  is  likely  to  grow.  The  thrust  to  introduce  cognitive  engineering 
should  be  intensified  and  not  abandoned. 
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Appendix  A 


List  of  In-Process  Deliverables 


Deliverables 

1 

Acceptance  Summary  Report 

2 

Accident  Risk  Assessment  Report 

3 

Acquisition  Decision  Memorandum 

4 

Acquisition  Information  Assurance  Strategy 

5 

Acquisition  Program  Baseline  (APB) 

6 

Acquisition  Strategy 

7 

Advance  Change/Study  Notice 

8 

Advanced  Concept/Joint  Capability  Technology  Demonstration  Proposal 

9 

Advanced  Concept/Joint  Capability  Technology  Demonstrations 

10 

Affordability  Assessment 

11 

Alternative  System  Review  (Customer  Needs  Review) 

12 

Analysis  and  Determination,  Benefit 

13 

Analysis  of  Alternatives  (Activity,  Briefing,  Plan,  Report) 

13b 

Analysis  of  Alternatives  Briefing 

13c 

Analysis  of  Alternatives  Plan 

13d 

Analysis  of  Alternatives  Report 

17 

Analysis  of  Material  Approaches 

18 

Analysis,  Behavior 

19 

Analysis,  Criticality 

20 

Analysis,  DOTMLPF 

21 

Analysis,  Fault  Tree 

22 

Analysis,  Level  of  Repair 

23 

Analysis,  Logical 

24 

Analysis,  Maintenance  Task 

25 

Analysis,  Reliability-Centered  Maintenance 

26 

Analysis,  Requirements 

27 

Anti-Tamper  Measures 

28 

Audit  Reports 

29 

Beyond  Low  Rate  Initial  Production  Report 

30 

Business  Case,  Logistics 

31 

Business  Case,  Open  Systems 

32 

Business  Modernization  Management  Program  Certification  Decision  Package 

33 

Capability  Development  Document 

34 

Capability  Production  Document 

35 

Certification  of  Compliance  with  the  Clinger-Cohen  Act 

36 

Certification  of  Compliance  with  the  Financial  Management  Enterprise  Architecture 

37 

Change  Control  Board  Minutes 

38 

Change  Control  Forms 

39 

Command,  Control,  Communications,  Computers  and  Intelligence  Support  Plan 
(C4ISP) 

40 

Command,  Control,  Communications,  Computers  and  Intelligence  Supportability 
Certification 
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41 

Communities  of  Interest  Definition 

42 

Communities  of  Interest  Identification 

43 

Competition  Analysis  for  Depot-Level  Maintenance 

44 

Component  Cost  Analysis 

45 

Component  Live  Fire  Test  and  Evaluation  Strategy  /  Report 

46 

Concept  of  Operations 

47 

Concept  Selection 

48 

Configured  Items 

49 

Consideration  of  Technology  Issues 

50 

Constraints 

51 

Contamination  Control  Plan 

52 

Contract  Change  Notice 

53 

Contractor  Cost  Data  Report 

54 

Contractor  Data  Requirements  List 

55 

Contractor  Selection 

56 

Contractor  Services  for  Operational  Plan 

57 

Cooperative  Opportunities 

58 

Core  Logistics  Analysis/Source  of  Repair  Analysis 

59 

Cost  Analysis  Requirements  Description  (CARD) 

60 

Cost/Schedule/Perfonnance  Trade-offs 

61 

Counterintelligence  Support  Plan 

62 

Critical  Path  Drivers 

63 

Critical  Program  Infonnation  List 

64 

Data  Access  Mechanisms 

65 

Data  Asset  Identification  and  Prioritization 

66 

Data  Management 

67 

DD  Form  1494  Spectrum  and  Electromagnetic  Environment  Effects 

68 

Defense  Acquisition  Executive  Summary 

69 

Deficiency  Solutions 

70 

Design  Change  Request 

71 

Design  Review  and  Audit  Plan 

72 

Design  Review  Data  Packages 

73 

Design  Review  Meeting  Minutes 

74 

Design  Verification  Report  (Requirements,  Verification  Plan,  Verification  Data) 

75 

Designed  Science  and  Technology  Information 

76 

Development  Test  and  Evaluation  Report 

77 

Discovery  Metadata  Catalogs 

78 

DoD  Component  Cost  Analysis 

79 

DoD  Infonnation  Technology  Security  Certification  and  Accreditation  [Process] 

80 

DOTMLPF  Change  Recommendation 

81 

Duration  of  Support 

82 

Early  Operation  Assessment 

83 

Earned  Value  Management 

84 

Economic  Analysis 

85 

Electronic  Warfare  Test  and  Evaluation 
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86 

EMC  Control  Program 

87 

EMC  Design 

88 

EMC  Test  Plans  and  Reports 

89 

EMC/EMI  Control  Plan 

90 

Engineering  Change  Order 

91 

Engineering  Change  Proposal 

92 

Engineering  Development  Models 

93 

Engineering  Job  Analysis 

94 

Engineering  Memorandum 

95 

Engineering  Order 

96 

Failure  Modes  and  Effects  Analysis 

97 

Failure  Report  (Root  Cause  Investigation) 

98 

Functional  Analysis 

99 

Functional  Area  Analysis 

100 

Functional  Block  Diagrams 

101 

Functional  Flow  Diagrams 

102 

Functional  Needs  Analysis 

103 

Functional  Requirements,  lower-level 

104 

Functional  Solution  Analysis 

105 

Global  Infonnation  Grid  Implementation 

106 

Hardware  Elements 

107 

High-level  Operational  Concept  Description,  OV-1  (Integrated  Architecture) 

108 

Human  Engineering  Program  Plan  (3-6.6) 

109 

Human  Systems  Integration  Strategy 

110 

Human-Machine  Interfaces 

111 

Independent  Cost  Estimate 

112 

Independent  Manpower  Estimate 

113 

Independent  Technology  Assessment 

114 

Industrial  Capabilities 

115 

Information  Assurance  Strategy 

116 

Information  Support  Plan 

117 

Information  Supportability  Certificate 

118 

Information  Technology  and  National  Security  Systems  Interoperability  Certification 

119 

Initial  Capability  Document 

120 

Initial  Operational  Test  and  Evaluation  Data 

121 

Integrated  Architecture  and  Supporting  Views  (list  these) 

122 

Integrated  Architectures 

123 

Integrated  Digital  Environment 

124 

Integrated  Logistics  Support  Plan 

125 

Integrated  Master  Plan 

126 

Integrated  Master  Schedule 

127 

Integrated  Support  Plan 

128 

Integrated  System 

129 

Integrated  Systems-level  EMC  Test 

130 

Integration  Requirements  Document 
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131 

Interface  Control  Documents 

132 

Interface  Definitions 

133 

Interface  Identification 

134 

Interface  Revision  Notice 

135 

Interference  Control  Requirements 

136 

Interoperability  Certification 

137 

Interoperability  Components 

138 

Interoperability  Requirements  Certification 

139 

IPT  Structure  (go  through  requirements  and  pull  these  out) 

140 

Job  Package  Authorization 

141 

Job  Tasks  with  Descriptions 

142 

Lessons  Learned 

143 

Liaison  Engineering  Orders 

144 

Liaison  Specification  Change  Notice 

145 

Life  Cycle  Cost  Estimation 

146 

Life  Fire  Test  and  Evaluation  Report 

147 

Life-Fire  Waiver  and  Alternative  Life  Fire  Test  and  Evaluation  Plan 

148 

Logistics  Plan  (see  also  number  125) 

149 

Logistics  Support  Analysis  Reports 

150 

Low  Rate  Initial  Production  Quantities 

151 

Maintainabilility  Demonstration  Report 

152 

Maintainabiility  Program  Plan 

153 

Maintainability  Demonstration  Plan 

154 

Maintainability  Plan 

155 

Maintainability  Prediction  Report 

156 

Maintainability  Status  Report 

157 

Manning  Documents 

158 

Manpower  Estimate 

159 

Manufacturing  Plan 

160 

Market  Analysis 

161 

Market  Research  Report 

162 

Mass  Properties  Control  Plan 

163 

Metrics,  KPPs,  MOEs,  MOPs 

164 

Mission  Analysis  Reports 

165 

Mission  Interface  Verification  Plan 

166 

Mission  Support  Plan 

167 

Modeling  and  Simulation  Validation 

168 

Modeling  and  Smulation  Plan 

169 

Models  and  Simulations 

170 

Modular  Open-System  Approach  (in  Acquisition  Strategy) 

171 

N2  Diagrams 

172 

Net-Centric  Data  Architecture 

173 

Net-Centric  Data  Guidance 

174 

Net-Centric  Data  Sharing  Plan 

175 

Net-Ready  Perfonnance  Parameter 
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176 

Operational  Assessment  Report 

111 

Operational  Requirements 

178 

Operational  Test  Agency  Report  of  Operational  Test  and  Evaluation  Results 

179 

Operational  Test  and  Evaluation  Report 

180 

Operational  Test  Plan 

181 

Operational  View  (Integrated  Architecture) 

182 

Operations  Interface  Control  Documents 

183 

Parts  Control  Program  Plan 

184 

Parts  Materials  and  Processes  Selection  List 

185 

Parts  Screening  Test  Matrix 

186 

Parts,  Materials,  and  Processes  Control  Plan 

187 

Performance  Budget  Document 

188 

Performance  Objectives  and  Thresholds 

189 

Performance  Requirements,  lower-level 

191 

Performance  Specifications 

192 

Performance-Based  Agreement,  Product  Service  Providers 

193 

Performance-Based  Agreement,  Product  Support  Integrator 

194 

Performance-Based  Agreement,  Product  Support  Providers 

195 

Personnel  Rosters 

196 

Post  Deployment  Regression  Testing 

197 

Post  Implementation  Reviews 

198 

Post  Independent  Analysis 

199 

Prime  Contract(Bundle  with  SOW  and  CDRLs) 

200 

Process  Design  and  Redesign 

201 

Product  Support  Plan 

202 

Product  Support  Strategy 

203 

Production  Plan 

204 

Program  Budget  Decision  Memorandum 

205 

Program  Deviation  Report 

206 

Program  Engineering  Documentation  Requirements  Notice 

207 

Program  Integration  Plan 

208 

Program  Objective  Memorandum 

209 

Program  Plan  (IMP) 

210 

Program  Protection  Plan  (Security) 

211 

Program  Requirements  List 

212 

Programmatic  Environment  Safety  and  Occupational  Health  Evaluation 

213 

Prototypes 

214 

Quality  Management  and  Control 

215 

Register  Metadata  with  DoD  Metadata  Repository 

216 

Registration  of  Mission-Critical  and  Mission-Essential  Information  Systems 

217 

Reliability  Estimate 

218 

Reliability  Plan 

219 

Reliability  Prediction 

220 

Reliability  Program  Plan 

221 

Request  for  Deviation/Waiver 
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222 

Results  of  Testing,  Experimentation  and  Evaluation 

223 

Review,  Critical  Design 

224 

Review,  Critical  Design  -  Subsystems 

225 

Review,  Defense  Acquisition  Board 

226 

Review,  Design  Readiness 

227 

Review,  Full-Rate  Production  Decision 

228 

Review,  Information  Technology  Acquisition  Board 

229 

Review,  Initial  Technical 

230 

Review,  In-Service 

231 

Review,  Integrated  Baseline 

232 

Review,  Operational  Test  Readiness 

233 

Review,  Physical  Configuration  Audit 

234 

Review,  Post-Deployment  Performance 

235 

Review,  Preliminary  Design 

236 

Review,  Preliminary  Design  -  Subsystems 

237 

Review,  Product  Support  Integrator  Perfonnance 

238 

Review,  Product  Support  Provider  Perfonnance 

239 

Review,  Production  Readiness 

240 

Review,  System  Functional 

241 

Review,  System  Verification  (or  Functional  Configuration  Audit) 

242 

Review,  Test  Readiness 

243 

Reviews,  Milestone  (A,  B,  C) 

244 

Risk  Assessment 

245 

Risk  Fist 

246 

Risk  Management  Plan 

247 

Risk  Monitoring 

248 

Roadmaps,  Architecture-view-based 

249 

Safety/Hazards  Analysis  Plan 

250 

Schedule,  Program  Development  (Integrated  Management  Schedule) 

251 

Security  Classification  Guide 

252 

Selected  Acquisition  Report 

253 

Service  Directory(s) 

254 

Software  Change  Request 

255 

Software  Elements 

256 

Software  Plan 

257 

Software  Reliability  Plan 

258 

Software  Resources  Data  Report 

259 

Software  Support  Plan 

260 

Solution  Sets 

261 

Specification  Change  Notice 

262 

Specification  Tree 

263 

Specification,  Configured  Item 

264 

Specification,  Prime  Item 

265 

Specification,  Segment 

266 

Specification,  Subsystem 
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267 

Specification,  System 

268 

Specifications,  Build-to 

269 

Specifications,  Design 

270 

Spectrum  Certification  Compliance 

271 

Standards  List 

272 

Statement  of  Work  (bundle  with  Contract) 

273 

Subcontract 

274 

Subsystem  Requirements  (under  Requirements) 

275 

Subsystems,  Hardware 

276 

Subsystems,  Human 

277 

Subsystems,  Software 

278 

Support  and  Maintenance  Effectiveness 

279 

Support  Environment  and  Locations 

280 

Support  Strategy  Review  Plan  Process 

281 

Survivability/Vulnerability  Plan 

282 

System  Maintenance  -  Support  Profiles  and  Use  Case  Scenarios 

283 

System  Requirements  (under  Requirements) 

284 

System  Requirements  Letter 

285 

System  Requirements  Review 

286 

System  Safety  Program  Plan 

287 

System  Security  Engineering  Aspects  Identification  and  Definition 

288 

System  Threat  Assessment 

289 

System  Transition  to  User 

290 

Systems  Engineering  Audit  Reports 

291 

Systems  Engineering  Plan 

292 

Target  Audience  Description 

293 

Technical  Performance  Management  Report 

294 

Technical  Performance  Measures 

295 

Technical  Standards  View  (Integrated  Architecture?) 

296 

Technology  Development  Strategy 

297 

Technology  Readiness  Assessment 

298 

Temporal  Analysis 

299 

Test  and  Evaluation  Master  Plan 

300 

Test  and  Evaluation  Strategy 

301 

Test,  Configured  Items 

302 

Threat  Assessment  Report 

303 

Total  System  Product  Support  Package  (with  Support) 

304 

Training  Materials  and  Devices 

305 

Training  Plan 

306 

Training  Programs 

307 

Transition  to  Government  Support  Plan 

308 

Unit  Cost  Report 

309 

Validation  Plan 

310 

Validation  Reports 

311 

Value  Engineering  Change  Proposals 
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312 

Vendor  Request  for  Information/Change 

313 

Verification  Memoranda 

314 

Verification  Plan 

315 

Work  Breakdown  Structure,  Contractor  (with  Contract?) 

316 

Work  Breakdown  Strcture  Dictionary  (with  Contract?) 

317 

Work  Breakdown  Structure,  Government 

318 

Work  Order/Work  Authority 
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Appendix  B 

Table  of  “Top  Ten”  (Actually  Nine)  Cognitive  Engineering  Activities 


Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

Field  Studies 

Cognitive  Task  Analysis 

Critical  Decision  Method 

Ethnography 

Surveys 

Questionnaires 

Interviews 

Cognitive  Task  Analysis 

Scenarios 

Environment  Characteristics 

SKAs 

Team  Dynamics 

Early  Operational  Assessment 
Operational  Assessment 

Operational  Testing 

Operational  Test  Agency  Report  of 
OT&E  Results 

Sustainment  Assessments 
Post-Deployment  Reviews 

Data  Asset  Identification 

User  Requirements 

Functional  Requirements 

HCI  Design  Specifications 

TES/TEMP 

Developmental  Test  and 

Evaluation 

Live  Fire  Test  and  Evaluation 

Test  Events 

Product  Support  Plan 

Training  Plan 

Beyond  LRIP  Report 

Full  Rate  Production  Decision 

Review 

User  Reviews 

Programmatic  Environment  Safety 
and  Occupational  Health  Evaluation 
(PESHE) 

Support  Strategy 
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Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

User  Scenarios 

Situation  Awareness  Oriented  Design 
Simulation 

Stop-Action  Scenarios 

Critical  Decision  Methods 

Scenarios 

New  Scenarios 

How  System  Is  Likely  to  Be 

Used. 

Scenario  Events  that  Draw  on 

Cognitive  Processes. 

Initial  Capabilities  Document 
Capability  Development 

Document 

Analysis  of  Alternatives 

Integrated  Architecture  Views 
Consideration  of  Technology 

Issues 

Data  Asset  Identification 

Design  Specifications 

TES/TEMP 

Developmental  Test  and 

Evaluation 

Live  Fire  Test  and  Evaluation 
Prototypes 

Metrics  and  Scenarios 

Models  and  Simulations 

Verification  Plan  and  Execution 
Validation  Plan  and  Execution 

Support  Strategy 

Walkthroughs 

Scenario  Review 

Information/Data  Flow  Review 

Inputs  From  Team  (Review)  of 

Design  Concepts  and  Artifacts. 
Consistency  and  Completeness 
Checks. 

Analysis  of  Material  Alternatives 
Analysis  of  Alternatives 

Integrated  Architecture  Views 
Consideration  of  Technology 

Issues 

Data  Asset  Identification 

Requirements  Analysis 

User  Interface  Specification 
(XML,  UML) 

Software  Specifications 

Operations  Concepts 

DoDAF  Products  (review) 

Product  Support  Plan 

Core  Logistics  Analysis 

Information  Support  Plan 

Root  Cause  Analysis 

Process  Design  and  Redesign 
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Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

Task  Analysis 

Cognitive  Task  Analysis 

Cognitive  Task  Analysis 

Logistics  Impacts  on  Availability 
Modeling  and  Simulation  Plan 
Definition 

Development  and  Re- 
Development  of  CONOPS 

Technology  Insertion  Impacts 

Locations  and  Resources  for 

Training 

How  System  Should  be  Operated 

Operational  Testing 

Operational  Test  Agency  Report  of 
OT&E  Results 

Sustainment  Assessments 
Post-Deployment  Reviews 

Integrated  Architecture  Views 
Consideration  of  Technology 

Issues 

Data  Asset  Identification 

User  Requirements 

Customer  Requirements 

Functional  Requirements 

TES/TEMP 

Developmental  Test  and 

Evaluation 

Modeling  and  Simulation  Plan 
Operations  Concept  (Hi-Fi) 

Product  Support  Plan 

Core  Logistics  Analysis 

Training  Plan 

Training  Materials 

Competition  Analysis  for  Depot- 
Level  Maintenance 

Support  Strategy 
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Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

User  Profiles 

Artifact  Study 

Contextual  Inquiry 

Contextual  Design  (Work,  Flow, 
Cultural,  Sequence,  Physical, 

And  Artifact  Models) 

Comparison  of  Population 

Description  used  in  Design  with 
Actual  Users  in  the  Field 
Determination  if  User 

Definition  has  Changed  as  the 

Result  of  a  Change  in  Usage 

Or  Technology  or  Concept 
Clarification  for  Walkthroughs 

Cross  Functional  Team 

Who  User  Is 

What  User  Is  Able  to  Do 

Identification  of  Needed 

Reasoning  Skills 

KSAs 

Population  Description 

Impacts  of  Career  Progression 

Through  Roles  to  Retirement 

SME  Usage  Preferences 

Operational  Testing 

Sustainment  Assessments 
Post-Deployment  Reviews 
Consideration  of  Technology 

Issues 

TES/TEMP 

Developmental  Test  and 

Evaluation 

Live  Fire  Test  and  Evaluation 
Modeling  and  Simulation  Plan, 

Design  and  Execution 

HCI  Design  Specifications 

Manpower  Estimate 

HSI  Plan  (may  be  in  SEP) 

Training  Plan 

CONOPS  (1) 

CWA 

Modeling 

Simulation 

Micro  Saint 

Naturalistic  Decision  Making 
Comprehensive  CTA  (focused  CTA 
won’t  do  because  doesn’t  give  end- 
to-end  picture  of  system) 

1.  Navigation  Model 

•  Bird’s  Eye  View  of  system 

•  Akin  to  a  site  map  for  a  web 
site  (a  system’s  site  map) 

•  How  information  navigates 
to  people. 

•  In  the  form  of  a  flow  chart. 

Operations  Concept 

Information  Support  Plan 

Integrated  Architecture  Views 
Command,  Control, 

Communications,  Computers,  and 
Information  Support  Plan  (C4ISP) 

Core  Logistics  Analysis/Source  of 
Repair  Analysis 

Human  Systems  Integration  Plan 
Metrics  and  Scenarios 

Models  and  Simulations 

Net-Centric  Data  Architecture 
Net-Centric  Data  Sharing  Plan 
Performance  Requirements 

Support  Strategy 
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Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

CONOPS  (2) 

“What  iffrng?” 

2.  Concept  Model 

Operations  Concept 

Qualitative  trade  studies 

•  This  is  the  concept  around 

KPPs 

which  you’re  developing  the 

HSI  Plan 

system. 

Critical  Operational  Issues 

•  Becomes  one  of  the  KPPs; 

User  Requirements 

may  result  in  more  than  one 

Functional  Requirements 

KPP. 

Joint  Capabilities  Document 

•  Driver  of  the  system,  why 

Initial  Capabilities  Document 

this  concept  is  being 

Capability  Development  Document 

developed  (vs.  driver  of  the 

Analysis  of  Materiel  Alternatives 

design-cost/sched/  perf) 

Analysis  of  Alternatives 

•  e.g.,  DDX:  Reduced 

DOTMLPF  Change 

manning  ->  quality  of  life 

Recommendation 

(habitability) 

Key  System  Attributes 
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Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

Training  Requirements 

Cognitive  Task  Analyses 

•  CTA 

•  ACTA 

•  CWA 

•  ACWA 

•  COGNET 

•  COSIMO 

•  Cognitively  Oriented  Task  Analysis 

•  Cognitive  Function  Modeling 

•  Concept  Mapping 

•  Contextual  Control  Model 

•  Course  of  Action  Analysis 

•  Critical  Decision  Method 

•  Decision  Ladder 

•  Decompose,  Network  and  Assess 
(DNA)  Method 

•  Empirical  Framework 

•  Goal-Directed  Task  Analysis 

•  GOMS 

•  Grammar  Techniques 

•  Hierarchical  Task  Analysis 

•  Hi-Lo 

•  Interacting  Cognitive  Subsystems 
Analysis 

•  KADS 

•  PARI 

•  RPD 

•  Semiotic  Models 

•  Skill-Based  CTA 

•  Sub-Goal  Template 

•  Task  Analysis  for  Error 

Identification 

•  Tasks  Analysis  for  Knowledge 
Description 

•  Task  Knowledge  Structures 

•  Team  CTA  Techniques 

•  Verbal  Protocol  Analysis 

Instructional  System  Design 

Behavioral  Task  Analysis 

Time-Motion  Studies 

Contextual  Design 

•  Interviewing  Techniques 

•  Knowledge,  Skills  and  Attributes 

•  Teaming  Profiles 

•  Descriptions  of  How  Knowledge 
is  Applied  to  Decisions. 

•  Contextual  Inquiry 

•  Work  Modeling 

•  Work  Redesign 

•  Consolidation 

•  User  Environment  Design 

•  Test  with  Customers 

•  Implementation 

Training  Plan 

Training  Requirements 

Training  Materials 

Support  Strategy 
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Integration  Points 

Methods 

Intermediate  Products 

Deliverables 

Feature  Definition 

Focused  CTA 

Descriptions  of  how  user  would  use  a 
designed  and  developed  system. 

Design  trade-off  studies. 

Group  dynamics  description. 

Design  Specifications 

•  User  interface  (HCI/HMI) 

•  Facilities 

•  Team 

Training  Requirements 

Consideration  of  Technology  Issues 
Critical  Operational  Issues 

Data  Access  Mechanisms 

Data  Asset  Identification 

Design  Readiness  Review 

Early  Operational  Assessment 

User  Requirements 

Full-rate  Production  Decision 

Review 

Independent  Technology  Assessment 
Interoperability  Requirements 
Operational  Assessment 
Post-Deployment  Performance 

Review 

Software  Products 

User  Reviews 
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Appendix  C 

Evolution  of  the  APSE  Interface 

Acquisition  Process  Support  vf 
Environment  (APSE) 


Ihl  f»  i  rrntrTfpn  •ir  APS?  .  •  tntan  i.#»  C  *tt»  "ufipn»t  Tijn‘  Hfyrj* v  rro^m  it 
U»tiQ  totted  by  Hit  viol kd  VjfUn  At<  Kn»  •••  (jontr  tit 

API!  wn  iittltd  ho  lauffl«f»  •*•<!  gmlt  Hu  iHMtnt  id  C ngntrrt  gngrotnug 
thojghtut  ttM  (#•  i fit  d  *  oonwlat  intat 


*««— — 


tMjrum 


Iml  wnai 


•  Me 


Figure  29:  37  Signals  Backpackit  interface 


Figure  30:  Concept  map  main  page 
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-  Deliverables  - 

▼  -  High  Value  Activities  -  ▼  -  Methods  - 

▼  -  Software  - 

■W 

APSE  Home 

Ask  the  Fxnerts 

Search: 

Supports 
Cognitive 
work  of 

/  Designs  Features  of 

/ _ / 


Acquisition  Practitioner  Support  Environment 

APSE  is  a  tool  that  supports  planning  and  day-to-day  practitioner  activities. 


Exerts  Control 
-Organizational 
—Budgetary 
—  Schedule 
-Staffing 
—  Contractual 


Improve 

High  Value  Activities \effectl',cnessof 


Operations  Concepts 

Measures  of  Goodness  |  Func 

JCIDS 

Concept  Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations 
&  Support 

Figure  3 1 :  Concept  map  with  high  value  activities  and  process  phases  at  bottom. 


•  A  need  of  opportunity  for  a 
new  capability  has  been 
identified 

•  That  need  has  been  described 
In  the  bioadest  terms  In  the 
US  DoO  this  is  done  in  the 
Jotrt  Capacities  Document 

•  Ideas  tor  non  materiel 
solutions  (doctnne 
urganl/dbon  training, 
leadership  and  education 
personnel,  and  or  facilities 
modifications)  have  been 
identified  Non  materiel 
solutions  do  not  require  an 
acqusibon  piogiam 

•  Ideas  fot  materiel  solutions 
have  been  identified 


.  A  set  ot  solution  approaches 
that  when  taken  separately  of 
together  appear  to  satisfy  the 
input  need  or  opportunity 


V. 


J 


V. 


J 


Figure  32:  Input  /  output  format  for  main  page 
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APSE  shows  where  big  returns  can  be  had  by  investing  in  human  performance, 
safety  and  satisfaction. 


Overview 


CMAP 


1 


DOD5000 


2 


Wturt  do  w*  do? 


High 
activities 


Next  generation  capabilities  require 
unprecedented  levels  of  performance  that 
can  only  be  achieved  by  amalgamating  new 
technologies  with  the  attributes  of  people 
who  will  design,  operate  and  sustain  them. 


APSE,  the  Acquisition  Practitioner  Support 
Environment,  facilitates  this  socio-technical 
synthesis  for  four  stakeholder  groups  - 
cognitive  engineering,  human  systems 
integration,  systems  engineering,  and 
program  management  practitioners. 


cepts  | 

Measures  of  Goodness  |  Function  Analysis  &  Allocation  | 

DOTMLPF  Analysis 

1  Project 

JCJDS 

Concept  Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations 
&  Support 

Figure  33:  Tabbed  main  page  with  selectable  overview,  concept  map  or  DAS  process 


101 


!  APSE  Home  -  Internet  Explorer  provided  by  Dell 


I  C:\Users\DCl\Documents\CognitrveSystemsEngineenng\aaaPhaseII\APSEDKign\ChrisBurnsAPSE 1002\newai  ~  ]  X  1  ■ 

ik  &  *  APSE  Home  e  Searched  for  'folder'  -  Cli...  &  APSE  Home 


■H 


Auto-  ■»  Send  to~  Q>  Settings^ 

V  ^  0  •  ~  •  E«ge  *  Tfiols  - 


r  AFRI^I 

APSE 

Ip  — -J 

-  Deliverables  • 


»  •  High  Value  Acbvibes  • 


■  Methods  - 


*  -  Software  - 


Search 


CD 

on 

CD 


APSE  supports  the  day-to-day  work  of  people  involved  in  a  product's  life  cycle. 
It  helps  developers  carry  out  the  tasks  that  have  the  greatest  impact  on  user 
performance  and  total  cost.  APSE  identifies  methods  and  software  aids  that 
help  system  developers  to  get  the  job  done. 

How  do  I  plan? 


Product 

Gov't 

Program 

Manager 

Contract 

Program 

Manager 

Systems 

Engineer 

HSI 

Analyst 

Cognitive 

Engineer 


Mouse  Over 

Initial 

Capabilities 

Document 


MouseOver 

Request 

For 

Proposal 


X 


X 


X 


X 


To  Do 


What  do  I  do? 

I 


Product  Pull  Down 


Key  Tasks 

♦PM 

♦SE 

•HSI 

•CE 

Manhours 

♦PM 

•SE 

♦HSI 

♦CE 

Methods  and  Procedures 

♦PM 

♦SE 

•HSI 

•CE 

Software  Aids 

♦PM 

♦SE 

•HSI 

•CE  ‘ _ 


JCIDS:  Customer  requirements  collected. 
Customer  need  is  identified. 


ro 

E 

E 

rs 

GO 


Multiple  rough  concepts  generated. 
Technical,  human  and  cost  implications 
explored. 

Technical  and  human  requirements 
stated  and  documented. 


U 

"O 

O 


re¬ 

s¬ 

ets 

E 

E 


Product  summary  text. 


D_  GO 


Our  Story 


What  will  give  me  highest  return  on  assets  employed? 


Figure  34:  PowerPoint  prototype  devised  after  conversation  with  Dr.  Fran  Greene 
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APSE  Welcome  Admin 


System  Development 

Integrated  Master  Plan  /Schedule 


WrdiwvMy,  Drcrmbrr  24,  2008 


WBS  /  IPT  /  CPT 
Pnme  Contract 
User  Interface 
Trade  Storfres 

Integrated  Master  Plan  /  Schedule 


se 

ss 

it 

u  oc 


n 

21 


E 

e| 

*  X 

w 

P  > 
>.  • 
(/>  O 


U 

5  E 


•o  w 


51 

3  O 


t  l 

O  * 

£  5 
a  V 
3  i 

in  ~ 


The  Integrated  Master  Plan 
iWP)  it  a  contractor 
prepared  customer -required 
tool  that  is  used  to  track  and 
measure  projecLTask 
accomplishment  It  identifies 
essential  events  program 
milestones,  event  entry  and 
exit  criteria  product  reviews 
techmcal  tasks  and  technical 
development  and  other  nsk 
reduction  activities  The  IMP 
consists  of  a  table  indexed  to 
the  Work  Breakdown 
Structure  and  a  descriptive 
narrative  of  execution 
processes  obgectwes  and 
control  documents  The  IMP 
is  an  event -based  plan  and 
contractually-binding 
document  that  is  relatively 
HUH 


GPU  Revww  and  approve  n 
defeerae* 


Key  Tasks 

•  pen  contract 


CPU  Oversee  prtperaPon  of  BIS  and  UP  and  approve 
Manhours 


Methods  4  Procedures 

GPU  Ooo^vnt  revie* 

CPU  Document  review 

SE  System  synthes*  storytoerdno  tn«Uie 

Software  Aids 

GPU  Mont 
CM  ucrosofl  Project 
S  E  Microsoft  P 


The  Integrated  Master 
Ptan  ts  a  contractual 
agreement  between  the 
project  owner  and 
contractor  Work 

descnbed  m  the  IMP 
cannot  be  de-scoped 
without  a  formal 
contract  modification 
Work  not  included  in 
the  IMP  will  not  be 
executed  H  all  system 
elements  -  hardware 
software  and  human  - 
are  to  be  considered 
human  systems 

mteorahon  and  coonitrva 


Mature  technologies  and  fit  aito  concept 


Figure  35:  DotNetNuke  main  page  designed  by  Brian  May 
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Appendix  D 

Software  Relevant  to  Embedding  Cognitive  Engineering  in  Systems  Engineering 


Software  Name 

Purpose 

ABRAHAM 

HSI  survey  tool 

ADVISOR 

Training  design 

AIM 

Training  material  development 

Altia 

Interface  development  and  rapid  prototyping 

AML 

Loosely  confederates  software  aids 

Analyst  Pro 

Requirements  management 

Artisan  Studio 

UML,  SySML,  and  DoDAF  Representations 

ASCENT 

Capture  of  constraints  and  cognitive 
requirements 

Belvedere 

Knowledge  capture  and  mapping 

BOAT 

Boxes  and  text  support 

•  Behavior  diagrams 

•  Data  and  control  flow 

•  Functional  flow  diagrams 

•  IDEF 

•  N2  charts 

•  Schematic  diagrams 

•  Signal  flow  diagrams 

•  State  charts 

Brahms 

Behavior  task  modeling 

CARE 

Requirements  engineering 

CD  Tools 

Gathering,  analyzing  and  sharing  qualitative 
field  data 

CogFIT 

Constructive  simulation 

CogniSystem 

Interpretive  structural  modeling  support 

COMET/VAMOSC 

Cost  analysis  from  HCDE 

Concept  Star 

Interpretive  structural  modeling  support 

CORE 

Requirements  management 

Cradle-5 

Requirements  management 

Create 

Facility  prototyping 

CSTD 

Navy-specific  workstation  design 

Delmia 

Ergonomics 

Distributed  Dynamic  Decision  Making  (DDD) 

Team  analysis  design 

DOORS 

Requirements  management  and  traceability 

EasyRM 

Requirements  management 

Envision  VIP 

Project  management 

Envision/Ergo 

Ergonomics 

ErgoMaster 

Ergonomics 

FAST 

Fatigue  analysis 

Foresight 

Modeling  and  Simulation 

Gatherspace 

Web  2.0  requirements  management 
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GRABIL 

Evaluation  of  interface  design 

HCDA 

Process  guidance  tools  from  HCDE 

iGen 

Embeddable  cognitive  agents 

IMAGE 

Function  characterization  from  HCDE 

IMPRINT 

Manpower  and  personnel 

Interchange  SE 

Data  backbone  and  analysis  interfaces  for 
integrated  design  environments 

IPME 

Human  performance  modeling 

IRqAR 

Requirements  management 

iSight 

Loosely  confederates  software  aids 

ISM 

Interpretive  structural  modeling  support 

Jack 

Ergonomics 

KollabNet 

Collaboration  software 

ManneQuinBE 

Ergonomics 

Micro  Saint  Sharp 

Behavior  task  modeling 

MindManager  Pro 

Project  Management 

Model  Center 

Loosely  confederates  software  aids 

MOST 

Team  design 

Objectivizer 

Requirements  engineering 

Process  Model 

Data  and  work  flow  modeling 

Project  Engine 

Project  management 

RETH 

Requirements  engineering 

RHEMS-D 

Human-machine  design  based  on  systems 
engineering  process 

Safework  Pro 

Ergonomics 

SALT 

Spatial  analysis  for  ergonomics 

SAMMIE  CAD 

Ergonomics 

Scenario  Plus 

Visualization  add-on  for  DOORS 

SEEC/Tiger  Pro 

Educational  tool  for  system  and  software 
engineering 

Ship-SHAPE 

HSI  analysis  (not  for  sale) 

SkillsNet 

Navy-specific  job  analysis 

Statestep 

Software  engineering  requirements  elicitation, 
specification  and  validation 

TacWISE 

Collects,  integrates  and  analyzes  perfonnance 
data 

Task  Architect 

Bookkeeping  of  task  attributes 

Taxonomic  Workstation 

Taxonomy  manipulation  (defunct) 

TIDE 

Organization  design  from  HCDE 

Total  Crew  Model 

Navy-specific  manpower  modeling 

WIBNI 

Freeware  requirements  management  database 

http :/  /www .  iawiki  .net/W  ireF  rames 

Wire  framing  useful  for  interface  prototyping 
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List  of  Symbols,  Abbreviations  and  Acronyms 


AMA 

Analysis  of  Materiel/Non-Material  Approaches 

AOA 

Analysis  of  Alternatives 

AHFEI 

Applied  Human  Factors  and  Ergonomics  International 

BOAT 

Boxes  and  Text 

ODD 

Capability  Development  Document 

CMAP 

Concept  Map 

CogEng 

Cognitive  Engineering 

CPD 

Capability  Production  Document 

DAG 

Defense  Acquisition  Guidebook 

DAS 

Defense  Acquisition  System 

DAU 

Defense  Acquisition  University 

DODAF 

Department  of  Defense  Architecture  Framework 

DOTMLPF 

Doctrine,  Organization,  Training,  Materiel,  Leadership  &  Education, 
Personnel,  Facilities 

FAA 

Functional  Area  Analysis 

FNA 

Functional  Needs  Analysis 

FSA 

Functional  Solution  Analysis 

HCDE 

Human  Centered  Design  Environment 

HCI 

Human-Computer  Interface 

HFES 

Human  Factors  and  Ergonomics  Society 

HSI 

Human  Systems  Integration 

HSIO 

[Air  Force]  Human  Systems  Integration  Office 

HSIS 

Human  Systems  Integration  Symposium 

HSIWG 

Human  Systems  Integration  Working  Group 

ICD 

Initial  Capabilities  Document 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IEWG 

[INCOSE]  Intelligent  Enterprises  Working  Group 

IMA 

Ideas  for  Materiel  Approaches 

IMP 

Integrated  Master  Plan 

IMS 

Integrated  Master  Schedule 

INCOSE 

International  Council  on  Systems  Engineering 

IPDT 

Integrated  Design  and  Process  Technology 

ISM 

Interpretive  Structural  Modeling 

ISP 

Infonnation  Support  Plan 

JCIDS 

Joint  Capability  Identification  and  Development  System 

KPP 

Key  Performance  Parameter 

MBSE 

Model-Based  Systems  Engineering 

MODAF 

Ministry  of  Defence  Architecture  Framework 

MOU 

Memorandum  of  Understanding 

PIA 

Post  Independent  Analysis 

PMI 

Project  Management  International 

QFD 

Quality  Function  Deployment 

SDP 

Structured  Design  Process 

SE 

Systems  Engineering 

106 


SEP 

SMCS 

SSD 

TBR 


Systems  Engineering  Plan 

IEEE’s  Systems,  Man  and  Cybernetics  Society 

Space  Systems  Division 

To  Be  Reviewed 
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